Marked Done Before It Was Done

Two posts ago, I wrote that I'd rather find the next payment bug in testing than in a client's money. This is the next one. It was found in testing, and it involves a client's money.

I've already written up the 80-second Stripe timeout the SDK defaulting to Node's https module, that module being shimmed under Cloudflare's nodejs_compat, requests hanging in the isolate's socket layer until an internal timer killed them. Two days to find, one line to fix; checkout now redirects in under 1.8 seconds.

That fixed the call going out to Stripe. This is about what happens after the money moves, which is a worse problem.

The handler I wrote first

typescript
const existingEvent = await WebhookEvent.findOne({ eventId: event.id });
if (existingEvent) {
  return NextResponse.json({ received: true }); // skip duplicate
}

await WebhookEvent.create({ eventId: event.id });  // mark processed BEFORE the work

await updatePaymentStatus(event);

It looks careful. It's deduplicating, which is exactly the thing you're supposed to think about with webhooks. Stripe guarantees at-least-once delivery, so duplicates arrive, and you have to handle them. I thought about it and wrote the check in the wrong order.

What that order does. updatePaymentStatus throws a Mongo connection spike, a validation error, and a cold start. The route returns 500. Stripe does exactly the right thing and retries with exponential backoff.

The retry arrives, findOne finds the event ID because it was written before the work that failed, and the handler returns 200 and exits.

The payment status is never updated. Not delayed, dropped, or permanently, with Stripe's retry mechanism cooperating, because the handler now reports success every time it's asked.

The money moved. The card was charged. My database says unpaid.

That's the worst class of bug I've written, and it's not because of the consequence. It's that the system is confidently wrong and reports success while being wrong. Everything downstream inherits it: the client's portal shows unpaid, my admin shows unpaid, and with real clients I'd have emailed someone asking them to pay an invoice they'd already paid. There's no alert for that. You find out when a customer tells you, which is after you've damaged the thing the portal existed to protect.

The record isn't a receipt; it's a claim

typescript
export interface IWebhookEvent {
  eventId: string;
  eventType: string;
  status: 'pending' | 'completed' | 'failed';
  error?: string;
  processedAt?: Date;
}

Three states instead of one boolean fact. The whole fix lives in that type.

Claim: if the event exists and is completed, it's genuinely done; exit early. If it's pending or failed, a previous attempt started and didn't finish, so re-execute. If it doesn't exist, create it as pending. Execute the work. Only then mark it completed.

When work throws now, the record goes to failed with the error attached, and the route returns 500. Stripe retries; the retry finds it failed, falls through, and runs the work again. The payment lands. The failure is recoverable because the record stopped claiming the work was done and started claiming it was attempted.

Three details that matter more than they look

$set, never $inc or $push. This is what makes re-execution safe: setting a status to 'paid' five times leaves it 'paid'; incrementing a counter five times does not leave it where it was. Idempotency isn't a property of the handler, it's a property of every individual write inside it, and one $inc breaks the whole guarantee while every other line looks correct.

Signature failures return 400, not 500. A bad signature means the payload isn't from Stripe or was tampered with, and retrying will never help. A 500 tells Stripe to keep trying, generating retry traffic forever for something that can never succeed. 400 says don't bother.

Use constructEventAsync, not constructEvent. Signature verification is a crypto operation synchronous on Node, async under WebCrypto on a V8 isolate. Same class of problem as the HTTP client: a library defaulting to a Node assumption that doesn't hold at the edge.

Why I'm still in soft launch

Two posts ago I explained that the site is live and I'm not signing clients' legal pages under attorney review, and an unwillingness to find the next bug with someone's money in it.

That was a principle when I wrote it. It's a receipt now.

The naive handler was merged and deployed and would have worked fine under every condition except one: a database hiccup at exactly the wrong moment. It might have run correctly for months. The first time it didn't, a client would have paid me and been told they hadn't.

All of this happened with test invoices and fake cards. That's the argument for the soft launch, and it's what I keep having to explain to people who see a live site with no clients and read it as a wasted month.

The full version is on Patreon. Everything essential is here.