Engineering Journal · ENGINEERING · May 13, 2025 · 16 min read
You Cannot Debug What You Cannot Reconstruct
When the bug will not come to your laptop, the work is not guessing. It is building a timeline from incomplete evidence until a next measurement is obvious.
By Golam Sorwar, Tech Lead and Full Stack Engineer in Dublin.
The useful production bugs are the ones that refuse to sit on your machine. They involve a tenant you cannot copy, a vendor who will not fail on demand, a race that needs last Tuesday's load, or a user who cannot remember the tap after the tap that mattered.
I used to fill that gap with confidence. Restart the worker. Clear the cache. Redeploy. Sometimes the noise stopped and we called it a fix. Sometimes the noise stopped because the world moved on. That is not engineering. That is weather reporting.
This is about the other habit: reconstruct what happened well enough that the next thing you look at is a question, not a ritual.
Context
The systems are the usual mix: Laravel, MySQL, Redis, queues, a payment or mail vendor, a mobile client. Logs exist. They are never the whole story. Metrics exist. They are usually about machines. The user is talking about a life: I paid, I did not get the letter, the list is wrong, it worked for my colleague.
You will not get a perfect dump. You will get a time, a person, a screen, and a feeling. That is enough to start if you treat it as the first layer of a timeline, not as a diagnosis.
I keep a short private list of questions that have wasted my time: which environment, which tenant, which binary if it is mobile, which job id, which provider id. If the ticket has none of those, the first engineering act is to ask, not to open the code. Opening the code feels like work. It is often a way to avoid the timeline.
The problem
The apparent problem is the symptom. It is slow. It is wrong. It failed.
The real problem is missing causality. You have effects and you do not have the chain. Local reproduction is a luxury that requires the chain to be portable. Many operational bugs are not portable. They are situated: this tenant, this job, this deploy, this payload.
If you skip reconstruction, you will "fix" a nearby thing. Nearby things are plentiful.
Production also withholds the replay button. You cannot always re-enter the same tenant, the same payload, the same vendor mood. So the work is not "make it happen here." The work is "make a chain that is good enough that changing the wrong thing would be embarrassing." If you cannot point at a row, a job, a deploy, or a vendor event, you are not debugging. You are renovating.
The tempting solution
Reproduce it locally, or keep changing production until the ticket goes quiet. Add logging after the fact in a panic, without knowing which question the new line answers.
Blame the vendor, the cache, the last deploy, the user. Those are hypotheses. They feel like conclusions because they are available.
Why that is not enough
Local environments lie by being clean. They have one tenant, one worker, yesterday's data, and your patience.
Restarting things destroys evidence. The queue you flushed may have been the only copy of the payload. The box you cycled may have been the only place the file landed.
A single log line without a request id, a tenant, a user, and a clock you trust is a postcard from nowhere.
Metrics that say CPU was high do not say which business action was in flight. You will scale the wrong thing. I have already written about that temptation with databases. It shows up here as a personality: when in doubt, reboot the graph.
Incomplete evidence is not an excuse to stop. It is a reason to be precise about what you cannot see. "We have no job id" is a finding. "We have no vendor event" is a finding. Those findings tell you whether the next step is a code change or an instrumentation change. Mixing them up is how you ship a patch that cannot be proven.
Options
Guess and ship a patch. Advantage: you look fast. Disadvantage: you will ship the patch next to the real bug.
Wait for it to happen again with more logs. Advantage: sometimes correct. Disadvantage: you are using users as probes.
Build a timeline first: user action, application logs, job logs, database row versions or updated_at, vendor dashboard, deploy history, feature flags. Then pick the next measurement that would kill the largest remaining hypothesis. Advantage: you spend time on questions. Disadvantage: it feels slow to people who want a restart. I want this to be the default.
A full observability platform before you have questions. Advantage: pretty. Disadvantage: you will still not look at it if the culture is restarts. Buy tools for the questions you already know you cannot answer, not as a substitute for a timeline.
Copy the record into a scratch environment and still fail to reproduce. Advantage: you tried. Disadvantage: you may have copied the row and not the job, the flag, or the vendor payload. Reproduction that ignores situated state is a second laptop lie. I would rather a timeline in production evidence than a staging clone that is missing the weather.
Trade-offs
You trade the comfort of "doing something" for the discomfort of not touching the box until you know what you would destroy.
You trade some user time. A careful investigation can be longer than a lucky restart. A lucky restart that erases the trail can be much longer.
You should not trade away a written timeline because it feels formal. Three bullets in the ticket are enough. Memory is a bad shared document. The next engineer cannot inherit your certainty. They can inherit your clocks.
Decision
I start with time. When did they see it. What deploy was live. What else happened in that window — a job backlog, a vendor status, a migration. Then I ask what evidence would have to exist if the popular hypothesis were true. Then I look for that evidence, or I note that we cannot see it, which is itself a finding.
I do not start with a change unless the system is on fire in the ordinary sense: we are losing money, leaking data, or cannot log in. Even then, the change is to stop the bleeding, and the timeline still gets written.
I also separate "we mitigated" from "we understood." Restarting a worker can be a mitigation. It is not a cause. If the ticket closes at mitigation, the next Tuesday will reopen it with less evidence. Write the residual risk in the ticket: we stopped the bleeding, we still do not know why the job stalled, here is the next measurement.
Implementation / Thinking process
Give every request and job a correlation id and put it on the logs you already have. This is dull and it is how reconstruction becomes possible next time.
Prefer facts that have clocks: row timestamps, queue available_at, provider event times. Align them. Clock skew is a character in this story.
Read the row. I still see people debug from the UI and never look at the record. The UI is a rendering. The record is closer to the crime.
Ask what the user could not see. Pending jobs, dead letters, a payment in captured-not-allocated. Those are the rooms the screen does not mention.
When you add a log, write the question above it in the pull request. If you cannot, you are hoarding string.
Compare clocks before you compare feelings. "It failed after deploy" is a hypothesis that needs the deploy time and the first bad event. I have watched people blame a release that landed after the first failure. The timeline would have saved the argument.
If the path crosses a vendor, open their evidence as a first-class source, not as a last resort. Their event id belongs on your timeline. If you cannot find it, that absence is a finding about your integration, which I have written about elsewhere as a neighbour problem.
I write hypotheses as sentences that can die. "The job never ran" dies if we find the job id. "The vendor never saw it" dies if their dashboard has the event. "The user double-submitted" dies if there is one intent. A hypothesis that cannot die is a prejudice. Prejudices ship patches.
Failure modes
The working hypothesis that you fall in love with. Disconfirm it on purpose. Ask what would prove you wrong.
The "cannot reproduce" close. That is a status of your laptop, not of the system.
The extra logging that includes a payload you should not store. Reconstruction does not outrank privacy.
The incident that becomes a personality story about a colleague. If your timeline needs a villain, it is not finished.
The patch that "might help" and closes the ticket. If you cannot say what evidence would have changed if the patch were right, you have shipped a ritual. Rituals accumulate. Evidence does not.
Operational consequences
Teams that reconstruct get calmer. They ask for ids instead of feelings. Support learns to send the time and the reference. That is a cheaper culture change than a new APM bill, and it makes the APM useful later.
Teams that reboot as a reflex train production to only fail in ways that survive a reboot. Those ways are nastier.
You will also change how tickets arrive. Once you ask for time, tenant, and a reference, people start sending them. That is not bureaucracy. That is how a timeline gets a first layer without an engineer excavating chat. If you never ask, you will keep receiving novels and keep opening the code to feel busy.
Lessons
Debugging without reproduction is historical work. You are not a mystic. You are trying to make the next observation more informative than the last.
If you cannot tell the story of the action, you are not ready to change the code. You are ready to change what you can see. A patch without a story is how a codebase collects souvenirs from weeks nobody can reconstruct. Souvenirs compile. They do not explain Tuesday.
What I would do differently today
I would have put correlation ids on jobs the same week we put them on HTTP. I treated background work as secondary. The bugs did not.
I would have written the timeline in the ticket even when I was sure. Sure is when you skip steps. The steps are for the sure days.
I would have treated "cannot reproduce" as a prompt to improve evidence, not as a close reason. If we cannot see it, the next commit is often a better clock or a better id, not a speculative if. Speculative ifs are how codebases fill with scars that never matched the crime.
Closing thought
A bug you cannot copy is still a fact in the world. Treat it like one. The laptop is optional. The clocks are not. Build the chain from the clocks you have. The restart will still be available when you know what you are restarting, and why, and what evidence you are about to delete.