Write It Down First

A shop owner messaged us on a Tuesday afternoon. Her customer was on the other line asking where his parcel was. Zagle said the order was in transit. The parcel had been delivered on Sunday.

We opened the courier’s panel. Delivered, Sunday, 4:12 PM. We opened our database. In transit. There was nothing in between the two. No error, no failed job, no record of a webhook arriving and not being handled. Pathao had done its part. It sent us the update, we returned 200, and then the update stopped existing.

Zagle runs a couple of thousand orders a month. That is not a huge number and I am not going to dress it up. But Zagle is multi-tenant. Every business on it runs its own storefront, some more than one, and picks its own couriers, at least one and usually two or three, out of Pathao, Steadfast and RedX, and we are adding more. An order that ships throws off eight to ten webhook hits over its lifecycle, one per status change, per courier, per business. So a month of shipments is somewhere between eight and ten thousand inbound calls, and each one carries a fact that someone is waiting on. Picked up. At the hub. Delivered. Returned.

Miss one and nothing turns red. The order just sits at a status that stopped being true.

The Missing Orders

Once I knew what to look for, I found the same thing in other places. It clustered around two conditions. One was load, when the VPS was busy and requests were slow. The other was deploys.

The signature was always identical. The webhook came in, the request completed, the courier got its 200, and the work that was supposed to follow never ran. Nothing in Sentry. Nothing in the logs, because there was nothing to log. The failure was the absence of a record.

Couriers do not help here. They fire and move on. Pathao might retry, Steadfast might retry, RedX might retry, and none of that is controllable or visible from our side. I cannot open a panel and ask what they sent me last Sunday. If we drop it in that window, the fact is gone. The parcel is still delivered in the physical world. Our copy of reality never finds out.

Five Files

I stopped guessing and read the code. There were five places. All of them were mine.

The webhook handler did the work inline.

// features/webhooks/service.ts:88
const body = await req.json();
await shipmentService.handleWebhook(body);
return NextResponse.json({ ok: true });

Nothing is written before the work starts. If the process dies inside handleWebhook, the event never existed. There is no record that Pathao ever called. Retry is entirely somebody else’s policy.

Then the dangling promises. Five of them, across order ingestion, orders and products.

// features/orders/OrderIngestionService.ts:189
await session.commitTransaction();

void fraudCheckService.evaluate(order._id);
void notificationService.sendOrderPlaced(order._id);

The transaction commits. The order is real. The response has already gone back as 200. Then the process gets killed and the fraud check and the notification do not happen, and nobody finds out, because from the caller’s side everything succeeded. The same shape sits at orders/service.ts:761 and 782, and at products/service.ts:896 and 1148.

Seventeen lines later in the same file, the audit log.

// features/orders/OrderIngestionService.ts:206
await session.commitTransaction();
await logAuditEvent({ type: "order.created", orderId: order._id });

The order is written inside the transaction. The record of why it exists is written outside it. Today that costs a gap in a log. It stops being a small cost the first time someone asks what actually happened to an order three weeks ago and the trail has a hole in exactly the place they are asking about.

The fourth was app/api/cron/sync-stock/route.ts, with maxDuration at 600, looping over every storefront in one HTTP request. A restart at storefront seven of twelve means five never synced. There is no resume point, and the summary at the end never gets written, so there is not even a record that the run was partial.

The fifth was that we deduped orders by externalOrderId at line 121 and never deduped events. A courier redelivering the same status ran handleWebhook twice. That is harmless right now, because applying delivered twice is applying delivered. It stops being harmless the first time a handler does something that is not safe to repeat, like decrementing stock twice or sending the customer the same SMS twice.

Five bugs, five files, five features. They were the same bug. Work that needed to happen got lost because nothing was written down before it started.

The Notifications Folder

Months earlier I had built features/notifications/ for a narrow reason. Order notifications were unreliable and I wanted them to stop being unreliable. So I wrote a NotificationJob document with status, attempts, nextAttemptAt, lastError and lastResults. A worker claimed jobs atomically.

await NotificationJob.findOneAndUpdate(
  { _id, status: "pending" },
  { $set: { status: "processing" } },
);

Capped retries, backoff, a terminal failed state so nothing spun forever. It worked. Notifications stopped disappearing. I shipped it and moved on to the rest of the product, because there was a lot of product left.

I want to be accurate about this part. The clean version of the story is that I forgot something clever and rediscovered it later. That is not what happened. I knew what was in that folder. I had written every line. I just thought of it as a notification system, because it was in the notifications feature and that was the extent of my relationship with it.

What actually happened is that I sat down with five open bugs and by the third one I noticed they were not five bugs. And there was already a working answer to that exact problem sitting in one folder, solving it for one feature.

The question stopped being how to fix the webhook handler. It became why this was not how the whole application worked.

The Door

The first layer is the smallest one and it is the one that stopped the losses. When a courier posts to us we verify the signature, insert a Delivery holding the raw body, return 200, and do nothing else.

// features/webhooks/service.ts (after)
const raw = await req.text();
if (!verifySignature(raw, req.headers)) {
  return new Response("bad signature", { status: 401 });
}

await Delivery.create({
  courier,
  receivedAt: new Date(),
  headers: pickSafeHeaders(req.headers),
  rawBody: raw,
  status: "pending",
  attempts: 0,
  nextAttemptAt: new Date(),
});

return NextResponse.json({ ok: true });

One insert between us and permanent loss. Everything after it can now fail as loudly as it wants, because the bytes are on disk and the work can be tried again. A receiving desk signs for a package before anyone opens it. The signature is not a claim that the contents are correct. It is a claim that the package arrived.

The temptation here is to be clever about duplicates. Hash the body, drop anything we have seen before. I nearly did it, then went and looked at what the couriers actually send. Pathao and RedX give us no unique event identifier. Steadfast does not send a timestamp at all. So a content hash is the only handle available, and a content hash cannot separate a duplicate delivery from a courier legitimately re-sending the same status after a real retry, which happens.

Drop that at the door and you have quietly recreated the failure you set out to remove, with a better explanation attached. So the door does not decide anything. It writes.

Deduplication belongs in the handler, which knows the shipment’s current status and can tell whether this transition is a no-op. That decision needs domain knowledge. The door has none, deliberately.

The Redis Detour

Everyone’s first instinct here is Redis and BullMQ. It was mine too. The tooling is good and the path is well worn.

It is the wrong answer for this problem, and the reason is narrow. If the queue lives outside MongoDB, then writing the order and enqueueing the follow-up work are two operations against two systems, with a gap between them. Crash in that gap and you have a committed order with no job. That is the dangling promise bug again, now with an extra container to run.

You spend a weekend adding infrastructure and rebuild the same failure one layer down.

Keeping the queue in MongoDB means the event is written inside the same transaction as the state change it describes.

await Order.create([orderDoc], { session });
await Stock.bulkWrite(stockOps, { session });
await OutboxEvent.create([{
  type: "order.created",
  aggregateType: "order",
  aggregateId: order._id,
  streamKey: `order:${order._id}`,
  payload,
  auditable: true,
  status: "pending",
  attempts: 0,
  nextAttemptAt: new Date(),
  processedAt: null,
}], { session });

await session.commitTransaction();

Either the order exists and the event to process exists, or neither does. There is no third state. Set that next to the void fraudCheckService.evaluate(...) above and the difference is not cosmetic.

MongoDB is a mediocre queue. It is a mediocre queue that shares a transaction boundary with my data, and that one property is worth more to me than everything BullMQ does better.

Two Collections

I almost used one collection for both. A raw courier webhook and an internally authored domain event have nearly the same fields. A type, a payload, a status, an attempt count, a next attempt time. Merging them looked like an easy simplification.

They are different things, and the difference is where they came from.

A Delivery is untrusted bytes from outside. We keep it verbatim, because when a shop owner says her order shows the wrong status, the only useful artifact is exactly what the courier sent, not our reading of it. It is evidence. It never gets replayed into the domain.

An OutboxEvent is a fact we wrote ourselves, inside a transaction, about something we know is true because we just committed it. It has a schema we control and it is replayable by design.

Merge them and you end up enforcing schema discipline on JSON whose shape three separate companies can change without telling you. The retention rules also pull in opposite directions. Raw deliveries are debugging material with a short shelf life. Our own domain events need to outlive the incident that produced them by years.

Two collections. The line between them is where untrusted becomes trusted, and I want to be able to point at that line.

The Whole Path

Layer six is the one I would have skipped a year ago. Durable ingress protects a webhook we received. It does nothing about a webhook that was never sent, and couriers do miss. A daily poll of every shipment sitting in IN_TRANSIT past its expected window costs almost nothing and closes the last path to a stale status.

The Lease

The claim is one atomic write.

const now = new Date();
const event = await OutboxEvent.findOneAndUpdate(
  {
    status: "pending",
    nextAttemptAt: { $lte: now },
    streamKey: { $nin: blockedStreams },
  },
  {
    $set: { status: "claimed", leaseExpiresAt: new Date(now.getTime() + 60_000) },
    $inc: { attempts: 1 },
  },
  { sort: { createdAt: 1 }, new: true },
);

The lease is what makes a kill mid-deploy survivable. If the worker dies holding a claimed event, nothing has to unwind it. The lease expires and the event goes back to pending on its own. The recovery path for a process that was killed is the passage of time.

The Sidecar

deploy.yml:99 runs docker rm -f. That is a SIGKILL. No drain window, no graceful shutdown, no chance for in-flight work to finish.

I am not going to soften the deploy script to protect the worker. Deploys should be allowed to be abrupt, and the system should be dull about it.

If the poller runs in-process, every deploy kills work mid-flight and produces a burst of recoverable failures. That is noise, and noise is how you train yourself to ignore your own alerts. So the worker is its own container in docker-compose.yml. It restarts on its own schedule, and because it lives in the compose file it is version controlled and reviewed. A host crontab is neither. Crontabs live on one machine and appear in nobody’s git history.

One Order at a Time

WooCommerce sends order.created and then order.updated a few seconds apart. If created hits a transient failure and goes into backoff while updated succeeds, they land in the wrong order, and the order state gets built on an update to something that has not been created yet.

There were three options and two of them are traps.

Global ordering means one bad event stops every event behind it. A malformed payload from one storefront halts the other eleven. No ordering at all means every handler has to be commutative, and I am not going to write that rule into a codebase several people will touch and then trust it to hold for two years.

So the events are ordered per aggregate, partitioned by streamKey, which is order:<id>. A blocked stream blocks exactly one order. Everything else keeps moving. The failure also becomes legible, because a stuck stream has a visible depth. Three events queued behind a blocked one on a single order is a specific thing you can find and fix.

The Replay Problem

This is the part where I was wrong, and I would have argued for it confidently.

Once the event log existed and was reliable, event sourcing looked obvious. Make the log the source of truth for order state, store only events, and rebuild any order by replaying its stream. History for free.

Then I worked through what happens when a handler is wrong. Not a crash. Just wrong. A courier status mapped to the wrong internal state, so a batch of orders shows returned when the parcels were actually delivered.

Under event sourcing you fix that by correcting the projector and replaying. Which means the order’s history is now different from what its history said yesterday, and nothing anywhere records that it changed. That is not a correction. That is an edit to the past with no trace.

When a shop owner asks why her order sat at the wrong status for two days, the useful question is not what the order looks like now. It is what we believed on Sunday at 4:12 PM and what we did with it. Replay destroys exactly that.

So I reversed it. The event log is not the state. Every state change we write carries a sourceEventId pointing back at the event that caused it, with a unique index on that field.

OrderStateChangeSchema.index({ sourceEventId: 1 }, { unique: true, sparse: true });

Events are traceable-to, not derivable-from.

The arrow direction is the design. The state change points at the event. The event does not regenerate the state.

That index does two jobs. It makes replaying an event a no-op, so the retry machinery cannot apply the same transition twice. And it turns the question I am most afraid of, did this event actually get applied, into a query.

await OutboxEvent.find({ auditable: true, processedAt: null });

That is a dashboard, not a replay. The unprocessed backlog is a number you can put on a screen and watch go to zero.

The Dead Letter

The retry ladder ends. After the configured attempts, an event moves to dead letter and stops.

I considered retrying the important events forever, on the reasoning that giving up on a delivery status is not acceptable. That reasoning falls apart on inspection. An event retrying every hour for three days is not applied. It is unapplied with a longer log. The only real difference between infinite retry and a dead letter is whether a person gets told.

A dead letter is not the system failing. It is the system saying it needs a human.

So the safety lives in the alert, not in the persistence. Dead letter creation pages Sentry immediately. The dashboard shows dead, stuck and retrying counts, plus the age of the oldest pending event, and it has exactly one button, which is replay. If the oldest pending event is four minutes old, things are fine. If it is nine hours old, something broke this morning and nobody noticed, which is the state all of this exists to prevent.

What You Cannot Undo

Most of the things that will read this trail do not exist yet. Reporting, dispute resolution, whatever we build next year. I am writing the trail before the things that read it, and the order is deliberate.

Every other decision in this design is reversible under pressure. Wrong backoff curve, wrong lease duration, wrong partition key, wrong handler boundary. Each of those is a refactor and a deploy.

Retention is not. If I prune events at ninety days and need them eight months later, the trail for that period does not exist, and no amount of engineering brings it back. So auditable gets set aggressively, on every event type anything downstream might plausibly care about, and those events do not get pruned. Over-declare now and narrow later. Narrowing later is free. Widening later is impossible.

Where I Would Start

If you are looking at a system with the same holes, the order matters more than the tooling.

Write the inbound request to your own database before you do anything with it. That single insert removes the entire class of loss where you never find out. Do it first, because it works even if nothing else is built yet.

Then put the queue in the same database as your data, so the state change and the job to follow it commit together. If the queue lives somewhere else, you have a gap, and the gap is the bug.

Then move the worker out of the web process, because your deploys will kill it and you do not want to be arguing with your own deploy script.

Then decide, today, which events you will still want in three years, and never delete those.

Everything else can wait until something hurts.