A retry is not the same as a reissue

A CrowdRelay confirmation email failed to arrive. The obvious operational response was to find the dead webhook delivery and retry it.

The retry returned HTTP 200. The delivery was marked as delivered. The email still did not exist.

Transport success is not business success

That was the useful clue: transport success and business success had become two different stories.

The original event was fan.confirmation_requested. When it was created, its payload contained the email address, policy version and one-time confirmation token. The worker attempted to deliver it to the verified n8n ingress. Earlier failures had moved the delivery into a terminal state.

Retention removed exactly what it should

CrowdRelay also has retention rules for secrets. Once an outbox event is terminal and no delivery is still pending, the worker removes confirmation, recovery, unsubscribe, claim and coupon tokens from the stored payload. Keeping a dead one-time credential forever would make operational history unnecessarily dangerous.

The retry assumption was wrong

The retention rule was correct. The retry assumption was not.

The administrative retry reset the old delivery and sent the same outbox event again. By then the event no longer contained confirmation_token. The ingress accepted the signed envelope and routed it to the confirmation branch. The branch rejected the incomplete business payload, but the ingress response still told CrowdRelay that the HTTP handoff had succeeded.

The final record looked healthy:

Retry and reissue are different operations

This is why “retry” needs a precise meaning. A retry repeats the same operation with the same identity and inputs. A reissue creates new business material because the previous material is expired, consumed or deliberately destroyed.

A concert announcement can usually be retried. A request to send a one-time token may not be replayable after retention. The old event remains valuable as audit history, but it can no longer perform the original job.

Create fresh business material

The safe recovery path is therefore a reissue. CrowdRelay creates a fresh confirmation token, stores its hash, emits a new outbox event with a new identity and lets the normal delivery pipeline handle it. The old event is not edited back into usefulness and a secret is not reconstructed from database fragments.

Reject impossible retries early

The retry endpoint also needs to defend this boundary. Secret-bearing event types should be retriable only while the required secret is still present. Once retention has scrubbed it, the API should return an explicit reissue_required conflict instead of producing a delivery that can only fail later.

An acknowledgement needs a meaning

The ingress contract needed tightening too. Accepting an envelope is not enough when CrowdRelay uses the HTTP response to mark the delivery complete. The response must represent the outcome the sender actually cares about, or the asynchronous branch must produce a separate durable acknowledgement that CrowdRelay understands.

Do not weaken retention to save a button

I am keeping the secret scrubbing. Weakening retention to make an administrative button convenient would solve the wrong problem. The operation should reflect the lifecycle of the credential:

This bug was not caused by a complicated algorithm. Every component did something locally reasonable: retention removed secrets, retry resent an event and the ingress acknowledged a valid request.

The failure lived in the meaning between them.

Distributed systems are full of buttons labelled Retry. Some of them should really say Create a new operation.