Background worker

BackendTier 2 · build and shipPhase 4 · Take money

Code that does slower work after the user's request has already been answered, so the user isn't kept waiting.

Some work triggered by a request is slow: resizing an image, sending email, generating a report. Making the user wait for it holds up the response and ties up the server. A background worker exists to do that slow work separately, after the request returns, so the user gets a fast answer and the heavy job still gets done.

Sending a receipt email takes a second or two. If checkout waited for it, checkout would feel slow. Instead, the backend responds "order placed" immediately and hands the email off to a background worker that does it moments later.

Workers handle anything that can happen slightly later: emails, receipts, image processing, report generation. The user gets a fast response; the slow work happens out of sight.

A junior PM, stuck

Support asked why receipt emails land about fifteen seconds after someone orders, and whether that is a bug we need to fix. The dev said "it is a background job" and pasted worker logs at me. I could not tell from them whether the delay is normal or something is actually failing.

The delay is the design working, not a bug. Checkout answers the customer immediately and hands the slow email off to a separate worker that sends it moments later, so the order never waits on the mail server. Here are the log lines for one order's receipt, the checkout side and the worker side. Read the 201 first, then follow the email.

Order o_5512 checkout and receipt-worker log, 13:02
checkout service
2026-03-06T13:02:05ZINFOcheckout POST /api/orders 201 created o_5512 in 140ms user=u_88231
2026-03-06T13:02:05ZINFOcheckout receipt_email job enqueued order=o_5512
receipt worker
2026-03-06T13:02:08ZINFOworker picked up receipt_email order=o_5512
2026-03-06T13:02:08ZINFOworker rendering receipt for o_5512 total=649
2026-03-06T13:02:11ZWARNworker smtp timeout sending o_5512, attempt=1
2026-03-06T13:02:11ZINFOworker retry scheduled for o_5512 in 10s
2026-03-06T13:02:21ZINFOworker smtp send ok o_5512 attempt=2
2026-03-06T13:02:21ZINFOworker receipt_email done order=o_5512

Click a step to see the lines it points at.

Filing the fifteen-second delay as a bug. The log shows the receipt sent on a retry after one SMTP timeout; the asynchronous design is why checkout stayed at 140ms, and the delay is the expected cost of that.
Promising an instant receipt in the UI copy. The email is deliberately off the request path, so "check your inbox now" will sometimes be wrong; "arrives within a minute" matches how the worker actually behaves.
Reading the failure-and-retry line as an error. A worker that retries and eventually sends is healthy; the line to fear is a job that fails once and stops with no retry, which is a silent missing receipt.
Not designing for silent failure. "Did it actually send?" is a real question; if the worker had exhausted its retries there should be an alert, not just a buried log line, so someone knows a customer never got their receipt.

Tell support: "Not a bug. The order is created in 140ms and the receipt is handed to a background worker; for order o_5512 the first send hit an SMTP timeout at 13:02:11 and the retry succeeded at 13:02:21, so it lands about fifteen seconds later by design. The one thing worth adding is an alert if a receipt ever fails every retry." You read a job's lifecycle and told a normal retry apart from a real failure.

"The confirmation email arrives a few seconds after I order" is by design, not a delay bug. Understanding this shapes expectations you set in the UI.
Work that fails in the background (an email that never sends) needs retry and visibility; "did it actually send?" is a real question to design for.

"Move the receipt email to a background job so checkout returns instantly."

Appears in Phase 4, Take money.