Retry versus reissue for one-time tokens in CrowdRelay

A CrowdRelay confirmation email never arrived. The obvious fix was to find the dead webhook delivery and retry it.

The retry returned HTTP 200. The delivery was marked as delivered. The email still didn't exist.

Delivered, but no email#

That was the useful clue: transport success and business success had diverged.

The original event was fan.confirmation_requested. When it was created, its payload contained the email address, policy version and one-time confirmation token. The worker tried to deliver it to the verified n8n ingress. Earlier failures had moved the delivery into a terminal state.

Secret retention#

CrowdRelay also has retention rules for secrets. Once an outbox event is terminal and no delivery is still pending, the worker removes confirmation, recovery, unsubscribe, claim and coupon tokens from the stored payload. Keeping a dead one-time credential forever would make operational history unnecessarily dangerous.

What the retry did#

The retention rule was correct. The assumption behind the retry was wrong.

The administrative retry reset the old delivery and sent the same outbox event again. By then the event no longer contained confirmation_token. The ingress accepted the signed envelope and routed it to the confirmation branch. The branch rejected the incomplete business payload, but the ingress response still told CrowdRelay that the HTTP handoff had succeeded.

The final record looked healthy:

Retry and reissue#

So “retry” needs a precise meaning. A retry repeats the same operation with the same identity and inputs. A reissue creates new business material because the previous material is expired, consumed or deliberately destroyed.

A concert announcement can usually be retried. A request to send a one-time token may not be replayable after retention. The old event is still valuable as audit history, but it can no longer do its original job.

Reissuing the token#

The safe recovery path is a reissue. CrowdRelay creates a fresh confirmation token, stores its hash, emits a new outbox event with a new identity and lets the normal delivery pipeline handle it. The old event isn't edited back into usefulness, and no secret is reconstructed from database fragments.

Rejecting impossible retries#

The retry endpoint also needs to enforce this. Secret-bearing event types should be retriable only while the required secret is still present. Once retention has scrubbed it, the API should return an explicit reissue_required conflict instead of producing a delivery that can only fail later.

What an acknowledgement means#

The ingress contract needed tightening too. Accepting an envelope isn't enough when CrowdRelay uses the HTTP response to mark the delivery complete. The response has to represent the outcome the sender cares about, or the asynchronous branch has to produce a separate durable acknowledgement that CrowdRelay understands.

Keeping retention#

I'm keeping the secret scrubbing. Weakening retention to make an administrative button convenient would solve the wrong problem. The operation should follow the lifecycle of the credential:

No complicated algorithm caused this bug. Every component did something locally reasonable: retention removed secrets, retry resent an event and the ingress acknowledged a valid request. The failure was in what those steps meant together.

Some Retry buttons, like this one, should really say Create a new operation.