Engineering Journal · ENGINEERING · January 18, 2024 · 16 min read
Every Retry Is a Decision About What Failure Means
Putting work on a queue does not make it reliable. A retry without a theory of failure just repeats the accident with better timing.
By Golam Sorwar, Tech Lead and Full Stack Engineer in Dublin.
The first time I treated a queue as a reliability feature, I was really treating it as a place to hide latency. The request returned. The user was happy. The job would "sort it out." Sometimes it did. Sometimes it sorted it out twice. Sometimes it sorted it out against a world that had already moved on.
I have written about email as an operations problem, and about vendors as neighbours. This is the narrower claim underneath both: moving work off the request is not the same as designing asynchronous processing. The interesting questions start when the first attempt fails.
A retry is not a mechanical courtesy. It is a statement. You are saying that the failure was temporary, that the work is still the right work, and that doing it again will not create a second truth.
Context
The systems I work on lean on jobs the way operational products always do: webhooks, documents, sync, notifications, imports. Laravel queues, Redis, workers. The user-facing app stays snappy. The real work happens in a process that can die, restart, and run again without asking.
That is a good shape. It is also a shape that makes it easy to confuse "we have a queue" with "we thought about failure." The first is a dependency. The second is design.
I am not talking about email volume here. Mail has its own operational essay. I am talking about the generic habit: import this, sync that, notify them, generate the PDF, call the bank. The same worker personality shows up in all of them — hopeful, under-specified, and slightly proud of the retry count.
The problem
The apparent problem is time. The request is too slow if we do the work now. So we dispatch.
The real problem is that the work now has a life of its own. It can run after the user has left. It can run after the record has changed. It can run after a colleague already did the thing by hand. It can run twice because the worker died after the side effect and before the acknowledgement.
If you have not said what those situations mean, the queue will pick a meaning for you. Its meaning is usually "try again," which is only correct for a subset of failures.
There is a quieter version of the same problem: the job that succeeds against a record that is no longer eligible. The worker did not fail. The domain did. A retry policy that only looks at exceptions will keep performing favours for ghosts — a cancelled booking, a withdrawn application, a person who is no longer in the cohort. Success is not the same as still being the right work.
The tempting solution
Set tries to 3, add backoff, maybe a failed_jobs table. Feel responsible. This is what the framework makes easy, and ease is how defaults become doctrine.
If the job talks to a provider, wrap the call in a generic retry helper. Timeouts become retries. 500s become retries. Unique constraint errors become retries. Everything looks like weather.
Why that is not enough
Failures are not one species. A network timeout might be safe to repeat if the other side is idempotent, and disastrous if it already created a customer. A validation error will not become true because you waited a minute. A lock timeout might want a retry. A "this person is not eligible" should not.
Ordering is another lie we tell ourselves. Many queues do not preserve the story you had in your head. A later job can run first. If your handler assumes it is the only writer, you will invent last-write-wins bugs that only appear when the queue is busy.
User expectation is the third gap. The UI said "we will email you" or "we are processing." That sentence has a half-life. A job that succeeds tomorrow morning may be correct in the database and wrong as a product.
Observability is the fourth. A growing failed_jobs table is not a strategy. It is a graveyard with a count.
The fifth is fan-out. One user action becomes five jobs. Four succeed. One dies. The UI is green because the request was. The organisation is not, because the fifth job was the one that talked to finance. Retries on the four will not help. You need a composition: either the work is one intent with steps, or the UI must not lie until the set is done. Most teams do neither and call it eventual consistency, which is a fine phrase for a bad afternoon.
Options
Do the work in the request and accept the latency. Advantage: simpler failure, the user is still there. Disadvantage: you couple their patience to a vendor. Sometimes still right for tiny, safe writes.
Fire and forget with retries. Advantage: fast to write. Disadvantage: you have outsourced meaning to a counter.
Intent first, then a worker that can conclude: done, retry, dead, or needs a human. Advantage: the job describes work against current truth, not a snapshot that aged in Redis. Disadvantage: you write more state. This is the option I want for anything that creates money, messages, or records another system will treat as real.
A workflow engine. Advantage: visibility. Disadvantage: a second product. I would rather have a boring intent table than a platform we cannot staff.
Synchronous for the first side effect, async for the rest. Advantage: the user sees the dangerous part finish. Disadvantage: you have split one intent across two clocks. Only do this if the first part is small and the second part has its own visible state. Otherwise you have invented a half-queue.
Trade-offs
You give up the fantasy that async is free. You buy a state model and some dashboards. That is cheaper than duplicate side effects, but it does not look like velocity in a sprint review.
You also give up some throughput. Checking "have we already done this?" costs time. Not checking costs reputation.
What you should not trade is a way for a human to finish or cancel the work. Some jobs will die for reasons the code cannot honourably guess.
Decision
I want every job that can hurt to answer three questions before it is allowed to retry: is this failure transient, is the work still the work, and is the next attempt safe if the last one partly succeeded.
If I cannot answer those, I do not want a retry. I want a dead letter and a person, or I want the work redesigned until the answers exist.
The payload should be an identifier, not a novel. Reload the record. The world may have changed. A job that carries a stale "please mark paid" is how you fight the user.
I also want a visible "we are still working" that can expire. If the UI said processing, there is a time after which the honest sentence is failed, not still in the queue. Leaving people in processing forever is how you get duplicate taps and a second job you then have to make idempotent under pressure.
Implementation / Thinking process
Classify operations. Reads and webhooks you control can often retry. Creates against someone else's API need an idempotency key you persist, or a lookup before create. Updates need a version or a "still in the state I expected" check.
Give up on infinite hope. A bounded retry with jitter is a courtesy to weather. After that, the job is a case. Cases need owners, not more backoff.
Log the conclusion, not only the exception. We retried because the provider timed out. We stopped because the person is no longer eligible. Those sentences are how the next engineer debugs at speed.
Do not share one queue and one retry policy across interactive and bulk work. I have made that mistake in mail. It is the same mistake in any mixed workload. Urgency isolation is part of failure design.
Timeouts are part of the decision. A five-second HTTP timeout plus three retries is a twenty-second lie if the provider already started the work. Align timeout, idempotency, and user-facing time or you will retry yourself into duplicates while the user is still watching a spinner on another device.
Dead letters need a human path that is not "look at the table when you remember." A count, an age, an owner. Otherwise failed_jobs is a museum.
I also want the job to be allowed to do nothing. A no-op that records "already complete" is a success. A throw because the row is gone is a different success if the domain says the work is obsolete. Treating every empty result as a retry is how you invent storms. The handler should have a vocabulary: done, obsolete, retry, dead. Four words. Most queues only taught us two: throw or return.
// A retry is allowed only after we know the last attempt did not finish.
if ($intent->isComplete()) {
return;
}
$result = $intent->perform();
if ($result->isTransient()) {
throw $result->asRetryable();
}
$intent->conclude($result);
Failure modes
Poison messages. One bad payload blocks a worker or retries forever and starves healthier work. Quarantine is a feature.
At-least-once delivery plus a non-idempotent create. You now have two invoices or two messages and a support conversation that starts with "the system."
Silent success. The job returns because an exception was swallowed. The queue looks healthy. The user does not.
Clock drift and delayed jobs that fire after a cancellation. If you do not re-read eligibility, you will do favours for ghosts.
Operational consequences
Once people trust the queue, they will put more of the business in it. That is fine if stuck work is visible. If the only view is a log file, you have built a basement.
On-call quality follows this. A retry storm looks like a vendor outage and may be your own uniqueness bug. If you cannot tell the difference quickly, you will restart workers as a personality trait.
There is also a product consequence. Once users learn that "processing" sometimes means "lost," they will tap again. You will then need the idempotency you skipped, except now you need it under load and under support pressure. Design the failure while the tap is still rare.
Lessons
A queue moves time. It does not create safety. Safety is a theory of failure that the code is allowed to execute.
If you cannot say what a second attempt means, you are not ready to have a first attempt out of the user's sight.
What I would do differently today
I would have added an intent row before I added the third retry. I used to treat persistence of "what we meant to do" as extra. It is the whole plot.
I would also have stopped copying tries => 3 from the last job. Defaults are how unlike work becomes one policy.
I would have named the user-facing expiry in the same ticket as the job. If the UI says processing, that sentence needs a clock. Without the clock, support invents one, and their clock will be kinder than the queue deserves.
Closing thought
Dispatching a job is easy because the framework wants to help. Designing the failure is the work. Every retry says what you think the last failure was. If you have not chosen that meaning, the counter will choose it for you, and the counter does not know your domain.