Context

The mail is transactional more than marketing, though the line blurs when you send a lot of "please do this" messages on a calendar. Users are students, staff, sometimes agents or parents. The content is ordinary and the timing is not: a reminder that is useful on Monday is noise on Thursday. A receipt that arrives twice is a finance incident. A blast that goes to the wrong segment is a trust incident.

The application is still a Laravel monolith with workers, Redis, and a provider in front of the public internet. We did not need a separate company to send mail. We needed the monolith to stop pretending mail was a side effect of a controller.

I still write code in this path. I also get the conversations when a mailbox provider starts treating us more coldly, or when a scheduled send overlaps a peak login window, or when someone asks why a message is "sent" in the UI and absent in the inbox. Those conversations are the architecture.

The real problem

The apparent problem is delivery. Did it send. Why did it bounce. Can we resend. Those questions are valid and they are not the first ones.

The deeper problem is that email is a distributed product with a hostile downstream. You do not control inbox placement, delay, or whether a student is using an address their institution already trained filters to hate. Your success condition cannot be "the provider accepted the MIME." That is a hop. The product condition is "the right person got the right meaning in a window that still matters, and we can prove we did not spam them."

A second deeper problem is coupling. If password resets, invoice mail, and a scheduled notice share one queue, one provider key, one rate, and one worker pool, then a busy notice day becomes a login incident. I have seen "just email them" take down the thing the email was supposed to support.

There is also a data problem dressed as a feature. People want opens and clicks because they are used to marketing tools. In an operational setting, tracking is evidence and it is also a privacy choice. If you collect it, you now have a store that will be asked to explain itself. If you do not, you will still be asked why someone did not see the message. You have to pick which ignorance you can live with.

Constraints

Volume is lumpy. Monthly totals hide the hour when a term reminder and a payment notice land together. The system has to survive the lump, not the average.

We cannot buy our way out of reputation. A bigger instance does not make Gmail like you. Sending faster can make them like you less. Throughput is a political problem with a technical surface.

Templates change. Legal language changes. A stored body and a live template will disagree if you are sloppy about what you persist at send time. Historical mail is a record. If you re-render history with today's template, you have rewritten the past.

Mobile and web clients will trigger mail from more than one place. Idempotency is not optional. The same user hitting save twice, or a worker retrying, is how you get the "why did I get this three times" thread.

And we cannot pause the institution to rebuild mail. Every improvement ships next to live sends. That is the same constraint as every other essay in this series, applied to a channel people still treat as plumbing.

Options considered

Keep mailables in the request, default queue, one provider. Advantage: the framework already knows how. Disadvantage: you will discover isolation during an incident. Risk: a template bug or a slow provider takes the user-facing request or starves other jobs. Maintenance is easy until it is not. Business impact is a quiet accumulation of "email is flaky."

Move everything to a dedicated ESP product and operate from their UI. Advantage: bounce handling and analytics come in the box. Disadvantage: your application becomes a passenger; scheduled institutional logic still lives with you. Risk: two sources of truth for who was emailed, and a vendor change that your domain cannot describe. Fine as the pipe. Poor as the brain.

Build an internal mail platform with its own service and datastore. Advantage: clean boundary, independent scale. Disadvantage: you now run a product. Risk: the boundary is fake if the monolith still renders every template and the service is only SMTP with extra steps. I would do this when the team and the volume justify a second deployable. I would not do it as a way of avoiding queue design.

Keep mail in the application, but give it a real domain: messages as records, queues by class, schedules as first-class work, provider as a witness, tracking as an explicit choice. Advantage: the organisation can ask the application who was emailed. Disadvantage: you write more than a mailable. Risk: the table grows without retention. This is the option that matched our team and our need to explain ourselves.

Decision

We kept email inside the product and stopped treating it as a notification helper. A message is a row with a purpose, an audience, a template version, a schedule, a provider id, and a state we own. Workers send. The provider reports. We do not collapse those.

I chose that because the questions we get are application questions. Who was in the segment. Why did this person get the payment mail twice. Can we send this at 09:00 local and not at 02:00 because a job ran early. An ESP can help you see opens. It cannot, by itself, know your domain.

Why not a separate service? Because the cost of a second system would have been another neighbour, and this series is already full of neighbours. A clear module, separate queues, and a table we can query got us the isolation we needed without a new on-call shape.

Why not send in the request? Because the user's click is not a good timeout boundary for a provider, and because retries do not belong in a browser refresh.

Implementation / Process

The request writes an intent. It does not talk to SMTP. That sounds small. It is the whole reliability story. The intent has an idempotency key for the obvious cases: the same receipt, the same reset in a short window, the same scheduled notice for the same person on the same day.

Queues are split by class. Fast, small, user-blocking mail — resets, the message that unblocks a login — does not sit behind a term reminder to tens of thousands. Bulk has its own workers, its own rate, and its own permission to be slow. Mixing them is how you invent a false emergency.

Scheduling is a clock you own. We do not enqueue thirty days of rendered mail on the day someone clicks "plan." We enqueue the decision, and we materialise near the send time, with a cap. If you materialise too early, you send stale facts. If you materialise too late, you miss the window. The compromise is boring and written down.

The provider is a witness. We store their message id when we have it. Webhooks update delivered, bounced, complained. Those states do not overwrite our purpose. A bounce is a fact about an address. It is not, by itself, a fact about whether the student is still enrolled.

Tracking is narrow. If we need to know whether an operational mail was at least accepted, we can know that without building a marketing suite. Click tracking in institutional mail should be a decision, not a default. Defaults become surveillance by accident.

Rendering is versioned. The body we sent is the body we sent. Templates can evolve for the next message. Support should not have to guess.

Suppression is part of the domain. Bounces and complaints are not only provider numbers. They are a list you consult before you are a good neighbour. Sending again because a job retried is how you turn a bounce into a reputation problem.

Deployment of workers is part of mail. A release that leaves old workers on an old template, or that restarts everyone in the middle of a bulk window, is an email incident. We treat it that way now.

Problems and failures

We once let a bulk product share a connection pool with interactive mail because the provider account was the same. Account is not isolation. Rate and queue are. The morning felt like "email is down." Email was busy.

Scheduled sends drifted after a timezone assumption that was true for the office and false for the audience. A reminder that is technically on time in UTC can still be rude. We had to make local time a first-class input, not a format.

Retries duplicated mail when the provider accepted a message and we died before recording the id. That is the same partial-success story as payments, wearing a friendlier protocol. Idempotency keys on our side are not enough if we cannot ask, or safely assume, what they already took.

Tracking tempted people. As soon as a number existed, someone wanted to manage humans with it. We had to say no to a few uses. A low open rate on a plain-text operational notice is not a moral fact about a student.

Template includes and shared layouts created a class of bug where a footer change went out in a blast that had been approved in a different season. Versioning the sent body was the fix. Reviewing bulk content like a release was the process.

I under-communicated backpressure. When we slowed a send to protect reputation, it looked like the system was broken. Operations need a sentence: we are sending, on purpose, at a rate the internet will tolerate.

Trade-offs

We chose delay over blast. That is worse for the person who wants the campaign out before lunch. It is better for the next hundred days of inbox placement.

We chose a larger mail module over a smaller mental model. New engineers have to learn that "send email" is not one helper. I think that cost is honest. A helper that hides a million messages is a lie.

We chose less tracking than a marketing stack would consider normal. We lost some charts. We kept a cleaner story about what we collect.

We did not choose a separate service. We accepted that a serious module in a monolith can be the right size. If the team and the failure modes grow, that decision can be reopened. It should be reopened because of an isolation need, not because services are fashionable.

Result

Mail is still a source of tickets. It is a less mystical one. We can usually say whether we accepted the intent, whether a worker sent it, what the provider reported, and whether we suppressed the address. That is enough to have an adult conversation with support.

The interactive path no longer competes as directly with the bulk path. Busy institutional days still hurt. They hurt in the bulk workers, which is where they should hurt.

I will not invent a delivery percentage. Providers already lie with averages, and our audience is not a lab. The result I trust is operational: fewer days when "email" meant "the product is stuck," and a clearer no when someone asked to dump a segment at full speed.

What I would do differently today

I would have split queues before the volume made the split feel dramatic. Isolation is cheaper when it looks slightly premature.

I would have persisted the sent body from the first bulk product, not after the first "what did we actually send" argument. History is cheaper to keep than to reconstruct.

I would treat provider account structure as architecture earlier: separate credentials and streams for mail that must never share a reputation fate. That is a procurement conversation and a technical one. I used to think it was only procurement.

Broader lesson

At volume, email is not a library. It is an operations problem with a template on top. The design questions are the same as any other serious outbound integration: intent before IO, isolation of urgency, evidence after the fact, and a rate that respects a world you do not own.

If you still talk about mail as a notification, you will keep putting it on the default queue. The default queue is where other people's urgency goes to die.

Closing thought

A million and a half messages a month did not make us an email company. It made it indefensible to treat mail as a helper. Write the intent. Split the work. Keep the body. Believe the provider only as a witness. The inbox is not your server, and it will not adopt your release schedule.

Context

Operational environments — a college back-office, earlier ERPs and commerce tools — are full of features that were requested as relief. If we had this screen, we would stop chasing. The screen arrives. Chasing continues because the exception, the approval, the export to finance, or the "we also have to tell so-and-so" was never in the ticket.

Engineering is close enough to hear the request and far enough to miss the rest of the morning. That distance is the subject.

I do not mean this as a speech about empathy. I mean it as a measurement problem. If you never watch the ten minutes after save, you will keep shipping nouns. The organisation will keep paying for verbs.

I have also seen the inverse: a large automation that removed a click and created a reconciliation. The click was visible. The reconciliation was a week. We counted the click. They counted the week. Both counts were honest. Only one was the job.

The problem

The apparent problem is adoption. They are not using it.

The real problem is incomplete work design. We implemented the noun they named and not the job they are measured on. The job might be get this person to a state finance will accept, or leave a trail for the next shift, or do the same thing for twenty people before lunch. A feature that handles one pretty case can add clicks to the twenty.

There is a second problem: we measure engineering done-ness and they measure leftover effort. Both graphs can be honest and still argue.

A third problem is the exception that was declared out of scope. Out of scope is fine if it is rare and named. It is fatal if it is how Tuesday actually works. Then the official feature is a brochure, and the brochure does not shrink the week.

The tempting solution

Build exactly what was asked, on time, with tests. Declare success. Offer training.

Or add more features around the unused one, hoping the island becomes a continent. You can spend a year paving a process nobody wanted to walk.

Why that is not enough

Training cannot fix a flow that is longer than the spreadsheet. People are rational. They will use the tool that gets the morning over.

Exceptions that remain outside the system will pull the happy path outside too. If the official screen cannot do the ugly case, the unofficial file will do all the cases, because switching costs.

Reporting that does not match the number a manager already defends will be ignored. They do not hate your chart. They hate explaining a second number.

Customer-facing polish that does not change staff load can still be worth it. It is a different product goal. Do not call it operational relief.

The tempting solution also hides in automation language. "Now it is in the system" can mean "now they type it twice: once for us and once for finance." That is a technically correct integration and a larger morning. If you do not watch the handoff, you will celebrate the API and miss the second keyboard.

Options

Keep shipping requested screens. Advantage: you are responsive. Disadvantage: you become a form factory. The factory can be busy and the factory floor still full of paper.

Sit with the job for a morning and implement the crossing, not only the noun. Advantage: you see the spreadsheet columns that are the real schema. Disadvantage: it is slower to start and politically harder, because it sounds like you are rewriting the request. You are.

Refuse work that cannot name the effort it removes. Advantage: focus. Disadvantage: you will be called unhelpful. Sometimes you should still do the small courtesy feature. Not every ticket is a process redesign.

Automate the leftover. Advantage: if you understood the leftover. Disadvantage: if you did not, you automate the spreadsheet. I have already argued for watching first.

Ship the crossing before the noun: the export finance already trusts, the exception path, the batch of twenty. Advantage: the week can shrink before the screen is pretty. Disadvantage: the demo is dull. Dull relief beats a beautiful island. I would rather show a dead spreadsheet column than a new empty menu.

Trade-offs

You trade some output count for outcome conversations. Leadership must tolerate a week where the ticket looks large and the demo looks small.

You trade a little purity. The first useful version might be an export that matches the spreadsheet, not a beautiful workflow. Matching the spreadsheet can be the bridge. Living on the bridge is the failure.

You should not trade away a way to do the exception in the system. Exceptions are the process. A happy path that cannot host Tuesday will not host Monday for long either.

Decision

I want a definition of done that includes a sentence from operations: this replaces X, except Y, and Y is handled by Z. If we cannot get that sentence, we are building a parallel. I will still sometimes build the parallel if the political cost of waiting is worse. I will not call it relief.

I also want a date to look again. A week after launch, ask whether the spreadsheet column died. If it did not, the feature is not late. It is unfinished. That follow-up is more useful than a satisfaction survey.

If operations cannot name the effort that dies, I ask them to show the morning. The morning is more honest than a ticket title. I would rather delay a sprint than decorate a spreadsheet they will still open.

Implementation / Thinking process

Ask what they do after they would click save. The after is usually the work. A letter that still needs a manual email. A status that still needs a finance file. A list that still needs a human sort because the filter is wrong.

Watch one ugly case, not only the demo case. The ugly case is where the spreadsheet wins. If you only watch the demo, you will pave the exception into a side channel and then wonder why the official path is empty.

If the feature creates a new report, ask which report it retires. If it retires none, you have added a number. Adding numbers is not the same as adding clarity.

Admin tools matter here. If staff cannot fix a miss without engineering, the feature has a hidden support cost. I would rather a guarded, logged correction than a clean UI that requires a developer to amend a row.

Batch matters more than beauty. A screen that does one record well and twenty records badly will lose to a spreadsheet every time. If the job is twenty, the feature is not done until twenty is cheaper.

Finance and letters are the usual leftovers. If those still happen in another tool, say so in the launch. People can accept a known handoff. They cannot accept a story that the process is "in the system" when half of it is not.

I also ask who still does the sorting. If a human is the query planner — exporting, filtering in a sheet, re-importing — the feature did not absorb the job. It added a station. Stations that do not replace the next station are how mornings get longer while the roadmap looks complete.

Failure modes

The shadow process that we bless by ignoring. It will set the quality bar.

The feature that saves a specialist five minutes and costs a junior twenty. We asked the wrong person what "smaller" meant.

The integration that creates the record in our system and not in theirs, or theirs and not ours. Someone still types. We called it automated.

The success metric that is "usage." Usage of a worse path is not a win.

The launch that added a status and not a queue. Staff still hunt the twenty in a sheet because the screen only knows one. You shipped a noun. They still have a job of twenty. The leftover is the product you refused to see.

Operational consequences

When work actually shrinks, you hear it as silence. Fewer pings. A spreadsheet that ages. That silence is easy to miss if you only listen for launch noise.

When it does not shrink, you get a second job: maintaining a feature and a workaround. That is how systems become hard to change. I wrote about that earlier as coupling. Here it is coupling to a process you did not finish.

There is a trust consequence. After two islands, staff stop asking for software. They ask for exports. That is a rational response to a factory that ships nouns. Winning them back is harder than watching the morning before the first ticket. The cheapest time to learn the crossing is before you pave the island.

Lessons

Software can be correct and still be a detour. The organisation does not owe you adoption for a detour.

Done means the leftover work has a name, an owner, and preferably a smaller size. If the leftover is "the same as before," you shipped a museum of the request. Museums are accurate. They are not relief. Relief is a smaller morning, even if the screen is uglier than the mock.

What I would do differently today

I would have put the after-save walkthrough in discovery, not in a disappointed week-two chat. The after-save is the requirement.

I would have been more willing to ship an ugly in-system exception path earlier, instead of a polished happy path that staff cannot live in. Pretty and unused is a kind of technical debt. It looks like quality in a screenshot.

I would have asked which report dies. If none dies, I would have said so in the launch, instead of implying that a new chart is a new truth. Two truths are work. Work that does not shrink is the subject of this essay.

Closing thought

A feature that works is a low bar. The bar that matters in operational software is whether the week got smaller. If you cannot say which minutes died, you are still counting tickets. If the spreadsheet is still the system, the ticket was a scene, not a change. Build the crossing, or admit you only built an island.

Context

The work sits in college operations, not in a lab. Tickets from students and staff. Policy questions that are usually the same and occasionally not. Attendance and exception patterns that look like data and feel like judgement. Draft replies that must sound like the institution, not like a model that has read the internet.

We already had software. That matters. This was not a greenfield "AI product." It was a set of humans using a back-office every day, leaving a trail of statuses, comments, reassignments, and overrides. The trail was incomplete, like all trails, but it was more truthful than the process poster on the wall.

I also had a team that could ship ordinary automation — queues, rules, templates — without a model. So the question was never "can we call an API." The question was "are we about to encode a fiction." If the fiction is official process, the model will learn the poster. Staff will keep doing the real job in side channels, and you will have automated the part nobody was struggling with.

The real problem

The apparent problem is volume. Too many tickets. Too many repeated questions. Too much time spent copying a policy paragraph. Those are real, and they are the problems that get budget.

The deeper problem is that the repeated work is not the same as the repeated decision. Two requests can share a form and need opposite outcomes because of a fact the form never asked for: a date, a visa constraint, a previous exception, a conversation that happened at a desk. If you automate the form, you punish the cases that were already expensive.

A second deeper problem is that organisations often want automation to settle an argument they have not had. Who is allowed to make this exception. What "late" means. Which queue owns a refund question that is also a records question. A model will pick a winner with confidence. That does not make the argument finished. It makes the argument harder to reopen, because now there is a system in the way.

I care about this because I have already seen non-AI versions. A status automation that closed tickets people still needed. A rule that routed everything with a keyword to the wrong team because the keyword was how students talked, not how departments were organised. Adding a model to that instinct does not make it wiser. It makes it cheaper to scale the mistake.

Constraints

We could not halt operations to run a clean study. People needed replies this week. Any method that requires a six-month freeze before a single rule ships will be bypassed by a well-meaning script.

Data was messy. Comments were half the decision. Some outcomes lived in email. Some overrides had no reason recorded because the person was trusted and in a hurry. If your method requires perfect labels, you will either invent them or give up.

Privacy and tone are not optional in this setting. Student and staff text is not a public corpus. Drafts can be wrong in ways that are legally and socially expensive. A routing mistake is repairable. A confident wrong policy sentence is a different class of harm.

The team was small. We could not stand up a research org. The method had to be something engineers and operations could run together: export, cluster, argue, write a rule, watch.

We also could not treat "the model said so" as an audit trail. If a decision affects a person, someone has to be able to say why without pointing at a probability.

Options considered

Automate the official process now. Encode the poster in code or in a prompt. Advantage: speed, a demo, a feeling of progress. Disadvantage: you automate the fiction. Risk: staff invent workarounds around your workflow, which is how shadow process is born. Maintenance becomes a fight between the system and the people who still know the job. The business gets a faster wrong.

Put a model on the live queue immediately as the router or the reply. Advantage: it will look clever on the first twenty typical cases. Disadvantage: you have no baseline for what "typical" even was. Risk: silent drift, prompt changes nobody reviewed, a tone problem you find from a complaint. Complexity is hidden, which is the worst kind. I reject this as a first step in anything that touches people.

Hire more staff and change nothing. Advantage: judgement stays human. Disadvantage: you also keep the repetitive part that was actually regular. Risk: burnout, and a later automation project that is more desperate. This is sometimes the honest answer for a season. It is not a strategy if the work is truly repeating.

Observe, then propose, then automate with an audit. Mine historical requests and the human actions that followed. Use a model to group and describe patterns, not to act. Humans accept, rewrite, or reject each pattern as a rule. Only then does software act, with logging and a way back to a person. Advantage: you automate a decision someone can still explain. Disadvantage: it is slower to the first demo. Risk: the history is biased — you will learn how people behaved under last year's constraints, including their bad habits. That risk is why validation is not optional.

Decision

We used the long way: real requests, historical outcomes, human decisions, model-assisted analysis, human validation, then rules or a narrow workflow, then monitoring.

I chose that because I wanted automation that operations would not have to fight. If they do not recognise the rule, they will route around it, and you will have spent engineering time making the official system less relevant.

I also chose it because it keeps the model in a role it is actually good at. Models are useful readers. They can say "these four hundred tickets look like the same judgement with different nouns." They are weaker as unattended officers of an institution. I am willing to use them to draft, to suggest, to cluster. I am not willing to let them close the loop on a person's case until a human has said the loop is boring enough to be a rule.

Why not skip analysis and "just use RAG on the policy"? Because policy documents are what the organisation wishes it did. The decision log is what it did. Those disagree more often than people who write policy want to admit. Retrieval can help a draft once you know which question you are in. It cannot tell you which question you are in if you have never watched the inbox.

Implementation / Process

The pipeline is ordinary on purpose.

Start with the real requests. Not a synthetic set. The last season of tickets, forms, attendance exception asks, the actual text people sent. If you cannot get that, you are not ready to automate. You are ready to guess.

Attach the human decision. What queue did it go to. Who touched it. What status did it end in. Did someone override. Did a second person reverse the first. This is ugly joinery. Do it anyway. A request without an outcome is not a training story. It is a fragment.

Let a model analyse, not act. Ask it to group similar decisions and to write the rule it thinks the humans were using, in language a staff member would recognise. Ask it to find the cases that look like the group but went the other way. Those contradictions are the product. They are where the unofficial fact lives.

Then sit with the people who did the work. This is the step engineering tries to skip because it does not look like shipping. Show them the proposed rule. Listen for the sentence that starts with "yes, except." That except is either a second rule, a missing field, or a decision you should not automate.

Only then write software. Sometimes the software is a template and a tag. Sometimes it is a router with a small set of features you now know matter. Sometimes it is a draft reply that a human still sends. Sometimes it is a checklist the model is not invited to. The model may drop out of the runtime entirely. That is a success if the analysis taught you the rule.

Ship with an audit. Who decided, or which rule, on which version, with which input. Keep a hold-out: a slice of traffic that still goes to people, or a review sample, so you can see whether the rule still matches the job. When it drifts, you do not "tune the prompt in production" like a hobby. You reopen the description.

Attendance and policy lookup fitted this more cleanly than I expected, once we admitted that the first job was to name the exception types people already used. Ticket routing was messier because tickets are a bag of leftover work. We automated the boring centre and left the bag.

Problems and failures

The history taught us some habits we did not want to keep. People had routed a class of tickets to a particular inbox because one person was kind, not because the function belonged there. A model will call that a pattern. It is a dependency. Validation is how you refuse to industrialise a kindness that was already burning someone out.

We tried to analyse too many workflows at once. The method only works if someone can still hold the exceptions in their head. One workflow, one season of data, one table of contradictions. Then the next.

Draft replies created a tone problem before they created a facts problem. The facts were often fine. The voice was generic in a way that felt cold, or warm in a way that felt fake. Staff refused the drafts, which was the correct response. We had to write voice rules like we write validation rules: short, testable, owned.

Some stakeholders wanted the live model because the analysis phase looked like delay. I had to say that a confident router without a described decision is not faster. It is a future incident with better branding.

We also under-estimated how much of the decision was in attachments and hallway context. The text field is not the request. If your data only has the text field, your rule will be a rule about text fields. Say that out loud so nobody thinks you have automated the job.

Trade-offs

We traded a dramatic demo for a smaller, owned rule. I would make that trade again. Drama in this domain is usually someone else's day going wrong.

We left humans in the loop longer than a vendor pitch would like. That costs staff time now. It buys you an automation they will not sabotage.

We accepted that some work will never be a rule. That is not a failure of nerve. It is how you keep the institution from pretending judgement is a classification problem. Automation should make the remaining human work more visible, not less.

Result

The useful result was not "we have AI in production." It was that a few high-volume decisions could be written in a sentence a staff member would sign. Some of those sentences became ordinary software. Some became better forms that asked for the fact people were already using. Some became a draft that a human still judged. Those are different runtimes of the same analysis.

Where we skipped the analysis and jumped to a helper, we got something staff treated like a novelty and then ignored. I take that as evidence, not as a reason to push harder on the model.

I will not invent hours saved. I will say the conversations with operations got more specific. We stopped arguing about whether to automate and started arguing about which except belonged in the rule.

What I would do differently today

I would have recorded reasons on overrides before I asked a model to find patterns. Garbage in is not a cliché when the garbage is the only signal of judgement. A required one-line reason on an exception is an engineering change that makes later analysis sane.

I would have kept policy retrieval out of the first routing experiments. Mixing "where should this go" and "what is the answer" in one prompt made both worse. Sequence the questions. They are different decisions.

I would also have been clearer with leadership that the first deliverable is a description, not a bot. If they cannot value a description, they will not value a rule, and they will not support you when the bot needs to be pulled back.

Broader lesson

Companies try to automate a process before they can describe the decision. Software then hardens a misunderstanding. AI makes that cheaper and more fluent. It does not make it wiser.

Use the model as an observer: read the requests, read the human actions, propose the pattern, wait for a human to claim it. Then automate the claim. Monitor it like you would monitor a payment state, because you have changed how the organisation treats a person.

If you cannot write the decision in a sentence you are willing to defend, you do not have an automation candidate. You have a wish, and a model that will help you scale the wish.

Closing thought

The order is the argument. Real requests, historical data, human decisions, analysis, validation, then a rule with an audit. Reverse that order and you will ship something that looks like progress from a distance and like contempt from the inbox. I would rather automate a boring truth than a confident story about how the work was supposed to go.

Context

The stack is ordinary: application logs, a host or container view, maybe a few Laravel pulse-style numbers, queue depths if someone bookmarked Horizon. Vendors have their own dashboards. The business has spreadsheets.

None of these is wrong. The failure is when we treat host green as a proxy for "the organisation can do Tuesday."

I also care about who can read the board. If only one engineer can interpret the charts, you have not bought observability. You have bought a private language. The useful test is whether a deputy can answer "is the work moving" without calling you. That is the same holiday problem as folklore, wearing a prettier UI.

The problem

The apparent problem is lack of tooling. We should buy APM.

The real problem is lack of questions. If you do not know what you would ask, you will buy charts that answer "how busy is PHP." That is a systems question. Operational software also needs: how many intents are stuck, how old is the oldest mail, how many payments are captured and unallocated, how many jobs died since noon, did error rate change after the deploy.

If those questions live only in a person's head, you have a green dashboard and a human SLI.

The problem is also attention. People will manage what the board shows. If the board shows hosts, they will manage hosts. If the board shows stuck letters and unallocated payments, they will manage those. Observability is therefore a choice about what the organisation is allowed to ignore. A green tile is a permission to look away.

The tempting solution

Turn on everything. Log every request. Alert on every 5xx. Watch CPU. Feel covered.

Or alert on nothing until it hurts, then alert on that forever. The next month you have a noisy room and a muted channel.

Why that is not enough

Request logs without business identifiers are a firehose. You will not drink from them when you are tired.

Alerts that are not actionable become decoration. If the response is "we knew, it does that," you have trained the team to ignore the next alert, which might be the one.

Traces are wonderful for a slow endpoint and silent about a job that never ran.

Vendor dashboards are not your product's health. They are a neighbour's opinion. I have written about neighbours. Observability includes not believing them alone.

Business events without technical context are the other half of the lie. "Letters sent: 400" does not say whether that is the cohort you meant, whether the oldest is six hours late, or whether the worker is chewing a retry storm. A count without age is a press release. Operations need the lag.

Options

Host metrics only. Advantage: cheap, familiar. Disadvantage: you will miss the quiet functional failures — the empty send, the stuck sync.

Full platform, every service traced. Advantage: you can follow a request. Disadvantage: cost and noise if you have not named the golden paths.

A short list of user-meaningful signals, plus enough technical detail to debug them, plus alerts that imply a next action. Advantage: the board talks about work. Disadvantage: you have to maintain the list as the product changes. That is the job.

I want the third. Buy or build tools in service of it, not the other way around.

There is a fourth option I see too often: copy a vendor's "golden signals" poster and stop thinking. Latency, traffic, errors, saturation are a good start for a request-shaped service. They are a poor finish for a system whose pain is a job that never ran, a webhook that never arrived, or a human queue that doubled while HTTP stayed pretty. Use the poster as a floor, then add the signals that match how the organisation fails.

A weekly human review of stuck work and no new tools. Advantage: you will learn the questions. Disadvantage: it does not page you at 2am. Use it to decide what to instrument. Do not use it as the only nervous system once you know the sentence you care about.

Trade-offs

You trade volume of data for quality of questions. Less pretty, more useful.

You trade some engineering time to emit events — payment.allocated, mail.accepted, tenant.context.missing — that are not features. They are how features stay honest.

You should not trade away privacy. Observability that stores message bodies "just in case" is a second product with a worse privacy policy. A trail can name the intent without quoting the letter.

Decision

I instrument the paths whose failure is an organisational sentence, not a graph. Login, money, the outbound voice of the institution, the jobs that feed those. I want age and count of stuck work, not only throughput.

I alert when a human should do something within a bounded time. I do not alert because a number is interesting. Throughput without age is how a busy worker hides a dead one.

I also want a distinction between "the process is up" and "the work is moving." Those can diverge for a whole afternoon. If the only page you have is the first sentence, you will reassure people while the second sentence is already false.

Implementation / Thinking process

Start from the last painful questions, even if you will not invent the stories around them. What did we wish we knew. Put a number on that. If you cannot name the number, you are not ready for a chart. You are ready for a sentence. Charts come after that sentence already has a name.

Put the same identifiers everywhere: tenant, user, intent id, provider id. Correlation is the difference between a log and a trail.

Separate saturation (the box is full) from correctness (the work did not finish). Both can be red. They are different on-call.

For queues, depth without age will lie. A deep queue that is moving is a busy day. A shallow queue whose oldest job is six hours old is a stuck day.

Review alerts the way you review code. If nobody can say the action, delete or rewrite. An alert is a user interface for a tired person.

I also want a weekly look at the questions we still cannot answer. Observability work that only happens during incidents is just debugging with a nicer bill. The cheap time is the quiet week when you notice that mail age is invisible, or that payment stuckness is a SQL query someone runs by hand. Put that query on a board. Give it a threshold. Then you can spend the incident on the cause instead of on inventing the question.

Do not forget the human layer. A comment that says "retried, still pending" is an event. If those comments never leave chat, your system of record for operations is Slack. I would rather a small status on the intent than a long thread you cannot query.

I pick a handful of golden paths and give each a sentence: login works, money can be explained, the institution can still speak, stuck work has an age. Then I attach the smallest signal that would falsify the sentence. That is the board. Everything else is optional decoration until those sentences are cheap to ask.

Failure modes

The vanity dashboard. Twelve charts, no owner, opened for visitors.

The alert that fires on deploy every time. You will ship less, or mute it, and you will deserve both.

The metric that used a cache and then you cached the metric. You are watching a memory of health.

The log level that was debug in production for a week and cost you a disk, or a bill, or a leak.

The business metric that was implemented as a cached page fragment. You are now observing the cache. When the fragment is stale, the board is calm and the office is not. That is the green-tile problem wearing a prettier name.

Operational consequences

When the board can answer "is the work moving," conversations with non-engineers get shorter. You can say the mail age is fine and the payment intent is not. That is a better sentence than "the server is fine."

When it cannot, engineering becomes a priesthood that interprets green. Priesthoods do not scale, and they do not take holidays well.

There is a quieter consequence: you will overreact to the things you can see and underreact to the things you cannot. A CPU spike gets a meeting. A growing pile of unallocated payments gets a shrug until finance writes. That is not because people are careless. It is because attention follows the dashboard. Build the dashboard you want the organisation to care about.

On-call changes when alerts imply actions. A page that says "mail age > 30 minutes, check the failed provider and the worker" is a job. A page that says "error rate +12%" is a riddle. Riddles get muted. Jobs get done. If you cannot write the action, you are not ready to page.

Lessons

Monitoring is having numbers. Observability is being able to ask the next question, including "can the organisation still do the thing we sold them."

Green is a colour. Understanding is a sentence. If nobody on the team can say the sentence without opening five tabs, you do not have a dashboard. You have a collage. Collages do not page. Sentences do, if you give them a threshold and an owner.

What I would do differently today

I would have defined stuck-work views before I added another host graph. I liked graphs. Operations liked unfinished letters.

I would have given each alert an owner in the same sentence as the threshold. Unowned alerts are how channels die.

I would have put age next to every queue depth on the first board, not after we learned that a shallow stuck queue is worse than a deep moving one. Depth is a vanity metric for work that can stall.

Closing thought

If the only thing your dashboard can say is that the process is running, you have a heartbeat, not a nervous system. A heartbeat is necessary. It is a poor narrator for a week of work. Instrument the work. Alert on the action. Leave the green tiles for the things that are actually up, and stop asking them to speak for the week.

Context

I have had to think about this in travel-platform records moving into a newer back-office, and in operational systems that absorbed years of exceptions stored as notes, extra columns, and "we always do it this way." The source is never as clean as the destination schema you drew.

The people who know what a blank field meant are not always still in the building. The source system may still be running. Dual writing sounds tidy in a meeting.

I treat that inventory as part of the estimate, not as a surprise in week three. If nobody can explain a status, that status is a research task. If two teams disagree about a duplicate, that disagreement is the mapping. Pretending it is a type-conversion problem is how you get a fast script and a slow year.

The problem

The apparent problem is ETL. Map columns. Run the job. Count rows.

The real problem is semantic gap. Status values that look like synonyms and are not. Currencies that were implied. Identifiers that were reused. Duplicates that were the same person and duplicates that were not. History that was stored as overwrite. Foreign keys that pointed at rows you are not allowed to create in the new world.

If you cannot say what "paid" meant in the old system, you cannot say it in the new one. You can only paint it.

The problem is also proof. Even a perfect mapping is a claim until someone who lives in the domain has walked lives and totals. A developer who stares at a diff is not that someone. They can tell you the script ran. They cannot tell you the new row still answers the old question. If you skip that person, you will discover them as a ticket after go-live.

The tempting solution

A one-shot import on a weekend. Default the blanks. Drop the rows that fail validation. Go live Monday.

Or keep both systems forever and tell staff to look at the old one "if something looks wrong." Something will always look wrong. You have built a habit.

Why that is not enough

Defaults invent history. A blank date that becomes 1970, or today, will flow into reports and look like a fact.

Dropped rows are people. They will arrive as tickets. You have not failed validation. You have postponed a conversation.

Duplicates that you merge without a rule will create a polite frankenstein: one email, two financial lives, a support nightmare.

A migration without a reverse path trains you to be brave. Brave is a bad mood for data.

Reconciliation that is only "row counts match" will miss the row that moved and changed its meaning. Counts are necessary and almost never sufficient.

Foreign keys make this sharper. The old world may have pointed at a person you are not willing to create in the new world — a deleted user, a merged company, a placeholder. If you invent the missing parent to make the import succeed, you have created a citizen who never existed. If you drop the child, you have erased a fact someone will ask for. Both are decisions. The script should not pick in the dark.

Options

Big bang, accept the mess, fix forward. Advantage: one cut. Disadvantage: you will not know what you lost until a human is angry.

Shadow run: write to the new store, compare, do not cut over until the disagreements are named. Advantage: you find the semantic gaps while the old system still works. Disadvantage: you run two writers or a lag. Worth it for money and identity.

Migrate in slices — one cohort, one year, one module — with a written definition of done for each slice. Advantage: blast radius. Disadvantage: a long period of split brain. Need a rule for which system is allowed to change which slice.

Do not migrate history, only open items. Advantage: less dirt. Disadvantage: the organisation cannot answer last year. Sometimes they can live with an archive. Sometimes they are legally or operationally unable.

Migrate the keys and the open items first, history later as a read-only archive. Advantage: go-live does not wait for every old note. Disadvantage: two places to look. That can be honest if you label the archive as archive, not as a second current system.

Trade-offs

You trade calendar time for fewer fictional defaults. That trade is correct if anyone will make a decision from the new rows.

You trade some completeness for honesty. An explicit "unknown" is better than a pretty value.

You should not trade away the mapping document. If the mapping lives only in a script, the next argument has no evidence. A comment in a pull request is not a document the office can read.

Decision

I want a written map from old meaning to new meaning, including the meanings we will refuse to guess. I want a reconciliation that checks those meanings on a sample a human can understand, and on the aggregates finance or operations already trust — totals, counts by status, "this person has this open item."

I do not want a go-live that is only "the job returned 0."

I also want a leftover pile with an owner. Unknowns that sit in a spreadsheet called misc will become the next source system. Name the pile. Decide whether it is archive, manual review, or out of scope. Out of scope is allowed. Unnamed is not.

Implementation / Thinking process

Inventory the source like you would inventory a dependency. What are the statuses. What is allowed to be blank. What is unique in theory and not in the table. What is a key that other systems already stored.

Build the mapping as data, not only as code. A table of old_status → new_status → confidence. The unmapped ones are the project.

Preserve the old identifier. You will need it when a human says "in the old system I was 4412." If you lose the join, you have failed the first support ticket.

Run the import so it can be repeated. Idempotent upserts keyed on the old id. The first run will be wrong. That is not a scandal. An unrepeatable run is.

Reconcile in public. Show the mismatches to the people who know the domain before you show a dashboard that says 100%. They will find the lie the count cannot.

Pick a handful of lives and walk them end to end: the person, the open items, the last payment, the last letter. Aggregates hide the frankenstein merge. A single walked life will not. If you cannot walk ten lives without a specialist, you are not ready to walk ten thousand.

Keep a diff you can rerun. "It looked fine on the day" is not a method. A query that still compares old_id to new meaning after cutover is how you catch the late correction someone made in the source while you were celebrating.

I treat unknown as a first-class value. A status you cannot map is not "the closest enum." It is a row in the leftover pile with a reason. Painting it as the closest enum is how you get a tidy import and a lying report. Tidy is not the goal. Explainable is the goal.

Failure modes

Character sets and trimmed spaces that split one person into two.

Timezones that shift a date across a term boundary. A "day" is a business object.

A unique constraint in the new world that the old world never had. The import fails or silently suffixes. Both are decisions. Do not let the database pick.

A rollback that restores the old application and not the old identifiers you already sent to a vendor. The outside world has moved.

A sample that was only the pretty records. The ten lives you walked were the ones someone already understood. The dirt is in the notes, the duplicates, and the status nobody can define. If the sample avoids those, the reconciliation is a brochure.

Operational consequences

Staff will compare screens for months. If you cannot explain a difference in one sentence — we mapped X to Y because Z — they will not trust either screen.

Auditors, if they exist in your world, will ask how you know. "The developer ran a script" is not an answer. The reconciliation output is.

Staff will also invent a third system if the new one cannot do last year's trick. That trick is often the reason the old dirt existed. If you migrate the rows and forbid the trick, you have not finished the domain. You have finished the table and postponed the argument.

Support will speak in old identifiers for a long time. If you cannot join 4412 to the new record in one lookup, you have made every conversation a research project. Preserve the join. Put it on the screen. That is not nostalgia. That is how you keep the human map when the schema changes.

Lessons

A completed import is a claim about meaning. Treat it like a claim. Evidence, sample, aggregate, leftover pile of unknowns.

If you cannot say what you would do if the new truth is wrong, you are not migrating. You are jumping. A jump can be survived. It cannot be explained to the person who lived in the old row. They will ask what it meant. You will want the map.

What I would do differently today

I would have reserved time for reconciliation in the first estimate, as a first-class phase, not as "testing." Testing sounds optional to people who are tired. Reconciliation is the product.

I would have refused silent defaults in the mapping. Every default is a policy. Policies get names. Unnamed defaults are how fiction enters a tidy import.

I would have insisted on a repeatable import before the first "real" run. A migration you can only do once is a stunt. Stunts do not get a second chance to be honest about the leftover pile.

Closing thought

The rows will move. They always do. The work is to keep the meaning in the cart with them, or to admit which meanings you left behind. Admission is a kind of correctness. Silence is not. Until you can prove the new record still answers the old question, you have copied a table, not migrated a truth.

Context

Operational products have many kinds of people: students, staff, finance, an operator of the platform, sometimes an agent. Multi-tenant products add a sharper line: this data is not yours. APIs add a third: the caller is a programme.

Laravel gives you gates and policies. They are only as good as the question you ask them. can('update', $invoice) is a different question from can('void after settlement', $invoice). The first is convenient. The second is the business.

I also include the machine actors. A job, a webhook, a personal token, a "system" user used for imports — they all perform work. If your mental model of security is only people at browsers, you will write a careful policy and then bypass it in a console command because the command felt internal. Internal is still a door.

The problem

The apparent problem is unauthenticated access. Lock the routes.

The real problem is authenticated access that is too coarse. Once you are "staff," a surprising amount of the building unlocks. Once you are "admin," the building has no interior walls. Attackers like that. So do tired colleagues, which is more common.

A second real problem is object-level confusion. You checked that the user is staff. You did not check that the student belongs to their tenant. The permission was correct. The world was wider than the permission.

A third is time and aftermath. Who may do the thing is not the same as who may undo it, who may see that it was done, and who must be told. Security that stops at the click will miss the export, the PDF, the job that retries the send, and the admin who "just looks." Those are the same work wearing different doors.

Coarse roles also hide the real question during review. "Is this person staff" is easy to tick. "May they void this after settlement, on this campus, after the letter went" is the work. If the code cannot ask the second question, the review cannot either. You will approve a correct permission on the wrong rule and call it done.

The tempting solution

Roles: admin, staff, user. Middleware on the prefix. Policies that repeat if ($user->isAdmin()) return true.

Hide the button in the UI and assume the API is only used by the UI. I have already argued against that. It is a security failure wearing a design failure.

Why that is not enough

Admin as a synonym for trusted is how privilege escalates without a villain. Every exception becomes a reason to widen the role instead of to add a capability.

Secrets in env files are necessary and insufficient. The leak I worry about more, day to day, is an export that was allowed because exports are a staff feature, and the export included a column nobody thought was sensitive.

Audit trails that only log login and not "who changed the thing that matters" will not help you explain a record. Application logs tell you that a 200 happened. Business audit tells you that a status moved.

"Everyone is a bit of an admin" is the cultural version of the same hole. It starts as kindness. It ends as a model where the only role that can finish the morning is the widest one. At that point your ACL screen is fiction. The real ACL is whoever was shown the override last Friday.

Options

Coarse roles forever. Advantage: simple ACL screens. Disadvantage: you will keep adding people to admin. The set admin is not a team. It is a blast radius.

Fine-grained capabilities from day one. Advantage: precise. Disadvantage: you will build an IAM product. A small team will then bypass it.

Capabilities on the few workflows that can harm — money, personal data, cross-tenant, irreversible letters — and coarse roles elsewhere, plus object checks that always include tenant and owner. Advantage: effort where failure is expensive. Disadvantage: two styles. I can live with two styles if the expensive one is sacred.

A single superuser account shared in a password manager. I mention it only to refuse it. Shared gods do not audit.

Permissions designed from the org chart. Advantage: familiar nouns. Disadvantage: the chart is not the work. The person who covers Friday is not on the chart as "approver." If you implement the chart, the Friday person will borrow a login. Borrowed logins are the oldest privilege escalation in operational software.

Trade-offs

You trade some speed of "just let me fix it" for a recorded capability. Staff will feel blocked. That feeling is information. Sometimes the block is wrong and the capability should exist. Sometimes the feeling is the point.

You trade a little performance for object-level checks. I will pay that. A cached "is staff" bit that skips the tenant predicate is how you get a fast leak.

You should not trade away an audit on irreversible actions to keep the table pretty. A pretty table that cannot explain itself is a liability with nice margins.

Decision

I want the permission question to be in the language of the workflow. May this actor perform this transition on this record in this tenant. If we cannot say that sentence, I do not want a policy that says update.

Admin bypass, when it exists, is a capability with a reason, not a blank cheque at the top of every method.

I would rather have an awkward extra click that writes "overrode because the approver is away" than a silent god mode. The click is annoying. The silence is how you lose the only story you had.

Implementation / Thinking process

Every query that returns a person or a payment has a tenant or an owner predicate that does not depend on the developer remembering. Scopes help. Tests with two tenants help more.

Tokens carry the same questions as people. A personal access token that can read everything the user can read is a portable user. Scope it. Expire it. Know how to revoke it.

Exports are features. They get the same authorisation as the screen, and they get a column list that is reviewed. "Select *" is a privilege.

For the dangerous transitions, write an audit row in the same transaction as the change. If the change commits and the audit fails, I want the change to fail. An audit that is best-effort will be missing on the day you need it.

Think about time. A permission that was true in the morning may be false after a role change, a campus move, or a revoked token. Caches of "is staff" that outlive those events are not a performance win. They are a delayed breach. If you cache authorisation, the invalidation sentence has to be as serious as the one I want for any other cache.

Think about files. A letter PDF sitting on a disk with a guessable name is an API whether you called it one or not. Storage paths get the same object checks as the route that created them.

I also write the negative test first on the expensive paths: a second tenant, a narrower role, a token without the scope, a record in a terminal state. If the suite only tests that the happy actor can proceed, you have tested the brochure. The brochure is not where the hole lives.

Failure modes

The IDOR that looks like a guessable integer in a URL. Policies that check role and not object.

The job that runs as the system user and forgets to impersonate a tenant. System is a role. It is not an excuse.

The CORS or cookie change that was "just for local" and learned to love production.

The debug route that dumps the user. I do not need to invent an incident. The design is the incident waiting.

The policy that checks the role and then loads the record without a tenant predicate, because "we already authorised." Authorisation without an object is a mood. The object is the rest of the sentence.

Operational consequences

When permissions match work, onboarding is easier. You can give someone "letters" without giving them "void invoice." When they do not, every new hire is an admin by week two because that is the only way to do their job. That is not a people problem. That is a model problem.

Security review becomes possible. You can ask "who can void" and get a list. If the answer is "admins, and also anyone who found the old route," you do not have a review. You have archaeology.

You will also change how exceptions are requested. If the only path is "make me admin," every exception widens the blast radius. If the path is "grant this capability on this class of record, with a reason," the exception stays a sentence. Sentences can be reviewed. God mode cannot.

Lessons

Authentication answers who you are. Authorisation answers what work you may do. The second is a business document implemented in code. If the document is "admins can update," you have given up on the interior of the product.

Most serious mistakes look like correct permissions on the wrong rule. The badge was fine. The work was wider than the badge, and the record cannot tell the difference until someone asks you to explain a Tuesday.

What I would do differently today

I would have forbidden isAdmin() as an early return in domain policies. It is a trap that feels like pragmatism.

I would have put tenant assertions in the few queries that hurt, with tests, before I wrote a long security guideline. Guidelines do not fail CI.

I would have treated exports and file URLs as routes with the same object checks as the HTML. A CSV is not a report. It is a bag of other people's lives. If the screen was careful and the export was "select *", the carefulness was theatre.

Closing thought

You can lock the door and still leave the rooms connected. Check the work, not only the badge. The rooms are where the week actually happens. The permission can be correct and the business rule can still be wrong, and the record will not know the difference until someone asks you to explain it.

Context

The products I work on are integration-heavy because the organisation is. Money leaves through more than one rail. Invoices have to exist in an accounting system someone else owns. Email has to leave through a provider that will judge your reputation for you. Identity sometimes sits with a campus or a vendor. Mobile clients consume the same facts with a release cycle you do not command.

None of that is optional sophistication. It is the minimum set of relationships a real operation already has. Engineering did not invent the bank. Engineering invented the idea that the bank could be a function.

The team is small, so every integration has a human who "knows it." That knowledge is part of the risk. When they are away, the vendor dashboard becomes a haunted house. When they are present, we under-invest in making the failure obvious, because they can still read the runes.

The real problem

The apparent problem is reliability of HTTP. Timeouts, retries, mapping. Those are real and they are the part we already know how to talk about.

The deeper problem is that you have imported another organisation's release process, legal constraints, and operational habits into yours. They will change a field. They will deprecate a scope. They will have an incident during your peak, not during theirs. They will define success in a way that does not match your finance calendar. You cannot code review that. You can only decide how much of your product is allowed to be attached to it.

There is a second deeper problem: silent success. Many integrations fail by lying. They accept a request and process it later. They return 200 and then send the real answer to a webhook you were not ready for. They persist a customer in a slightly different shape than you sent. Your user sees "done." Your other system sees nothing. That gap is where operations invent spreadsheets.

I have also seen the cultural version. We treat an integration as finished when the demo works, because the demo uses a sandbox that is friendlier than production. Sandbox is not a smaller production. It is a different product. If your design only works there, you have a prototype with a logo.

Constraints

You cannot not integrate. Telling a college to stop using a bank, or a finance team to stop using their ledger, is not architecture. It is theatre.

You do not control their uptime, their versioning manners, or their support queue. SLAs, when they exist, are not the same as your user's patience.

Data has to remain explainable after a failure. "We will replay" is easy to say and hard to do if you never stored the inbound payload, the decision you made, and the external id.

Clients of your own API — including a Flutter app — will keep old behaviour. If your reaction to a vendor change is to change your public shape immediately, you have multiplied the blast radius.

A small team cannot wrap every vendor in a perfect anti-corruption layer on day one. If the standard is perfection, people will skip the layer and "just call the SDK" in a controller. The standard has to be the minimum that keeps failures diagnosable.

Options considered

Call the vendor from the request that needs it. Advantage: simple to write, easy to demo. Disadvantage: your user's latency is now their latency, and your error page is now their outage. Risk: retries in the request double-charge or double-create. Maintenance looks small until the afternoon an endpoint hangs. Business impact is immediate and public.

Put every vendor behind a full internal platform. Advantage: one language, one set of dashboards. Disadvantage: you have built a product for yourself that needs staff. Risk: the platform lags the vendor and people bypass it. Complexity is high. I have sympathy for this in a large org. I have suspicion for it as the first move in a small one.

Isolate each integration behind a narrow module, async where the user can wait, with stored intents and stored evidence. Advantage: you can retry without improvisation; you can show a human what you sent and what came back. Disadvantage: more moving parts than a controller. Risk: jobs pile up and you have only moved the outage into a queue. That is still better if the queue is visible.

Buy a unified "integration layer" product. Advantage: someone else maintains connectors. Disadvantage: you have added a fourth organisation to a three-organisation problem. Risk: mapping becomes a black box you cannot replay. Sometimes worth it. Often a way of postponing the moment you understand your own money.

Accept the vendor as the UI. Deep-link staff into Stripe or QuickBooks and keep your app thin. Advantage: less code. Disadvantage: you have trained the company to live in someone else's product, and you have no single story. I will do this for rare admin actions. I will not do it for the daily path.

Decision

We treat integrations as untrusted neighbours. The application is allowed to ask them to do something. It is not allowed to forget what it asked, or to assume the neighbour's status language is our product language.

That means an internal record first, a vendor call second, a recorded result third. Where the user can wait — email, invoice sync, non-blocking confirmation — the vendor call leaves the request. Where they cannot wait, we still write the intent before we talk to the network.

I chose that because I was tired of incidents whose only artefact was a stack trace and a memory. If you cannot answer "what did we send, what did they say, what did we conclude," you do not have an integration. You have a hope.

Why not a grand platform? Because a small team needs isolation more than it needs a brand. A module, a table of evidence, a job, a dashboard of stuck work. That is enough to stop the neighbour's worst day from becoming an unsearchable myth.

Implementation / Process

The pattern is repetitive on purpose. For each vendor we name the operations we actually use, not the ones in their marketing. Create payment. Refresh token. Post invoice. Send batch. Then we store an intent with our own id, send their id back into that row, and keep a short log of payloads we are allowed to keep.

Auth is infrastructure. Token refresh that lives in three jobs will fail in the fourth. Centralise it, encrypt it, alert when refresh fails, and do not log the secrets you just rotated.

Timeouts and retries are specific. A read can retry. A create cannot, unless you have an idempotency key they honour or you can safely ask "do you already have this?" before creating again. I have seen well-meaning retry middleware invent duplicate customers in an accounting system. The middleware was proud. Finance was not.

We map their states to ours in one place. The rest of the application is not allowed to switch on a vendor string. That sounds fussy until the vendor adds a state that means "successful but not really."

Observability is about stuck work and unknown states, not only error rates. A 99% success rate can still hide the ten invoices that did not sync, and those ten are the month.

When we add a new rail — another bank, another mail sender — we copy the pattern, not the old controller that "already did something like this." The second integration is how the first one's shortcuts get promoted into a standard.

Problems and failures

We stored too little, then too much. Too little and you cannot replay. Too much and you have a pile of personal data in a debug table that nobody thought was a datastore. Retention belongs in the design, not in a later panic.

We trusted sandbox field names. Production added a required field that the sandbox never enforced. The first real customer of that path became the test. That is an integration testing failure and a humility failure.

Partial success toward two vendors in one job created a repair genre. If you post to accounting and then fail to update your own row, you will do it again, or you will think you did not do it. Split the work. Make each step restartable.

Operations found vendor dashboards faster than they found our admin, because our admin was shy about showing the evidence. If the human still has to leave your product to understand your product, the isolation layer is unfinished.

I under-communicated change windows. A vendor deprecation arrived in an email that looked like marketing. Someone has to own reading those, and it cannot be "whoever noticed." That is a process control, not a code control.

Trade-offs

We write more code than the SDK sample. That is the fee for being able to explain yourself. I will pay it for money, identity, and anything that emails thousands of people. I will not pay it for a one-off admin import.

We accept delay. Async is safer and less charming. Product has to be told that "sent" may mean "accepted by us" and not "accepted by the bank." If you cannot have that conversation, you will lie in the UI.

We accept that some rare actions stay in the vendor UI. That is a concession to team size. It is also a concentration of risk in those actions, so they should be rare on purpose.

Result

Incidents did not disappear. They became shorter to narrate. We could usually find the intent, the external id, and the last response, which changes the call with support from theatre into a ticket they can work.

New integrations still hurt. They hurt in a known way. That is the most I will claim. Anyone who promises that a pattern removes vendor risk is selling you a repository again.

The organisational result was a slightly more adult conversation about adding "just one more" provider. The cost could be described as another neighbour, not another afternoon.

What I would do differently today

I would have given deprecation mail a named owner on day one. Code ownership without inbox ownership is how you wake up on a deadline the vendor set two months ago.

I would have built the stuck-work view before the third integration, not after we already had folklore. The first integration teaches you the pattern. The second will copy your omissions.

I would push harder on not sharing one job across two vendors. Every time I allowed it for speed, I bought a repair tool I did not have time to make nice.

Broader lesson

Your repository is the part of the system you can diff. The risk is often in the parts you can only log. Design for evidence, replay, and a language that belongs to you.

If an integration cannot be explained without opening their dashboard, you have extended their product, not yours. Sometimes that is fine. Call it that, so you know what you have chosen.

Closing thought

Add the provider if the organisation already depends on them. Then spend the rest of the design on the day they change, delay, or lie in a polite JSON body. That day is not an edge case. It is the reason the integration exists in your architecture at all.

Context

Small product groups — web, API, a mobile client, operational software — grow this dependency naturally. The first person who wired payments or mail or the tenant switcher keeps getting the ticket. Review becomes a bottleneck because only they can smell the danger. Documentation feels like a luxury when the same person is also shipping.

I still write code. The trap is using that as proof that the knowledge is shared. Code that only one person dares to change is not shared. It is a private language with a public repository.

The same pattern shows up in vendor access. One person has the Stripe login, the mail provider, the DNS, the "real" AWS console. When they are away, the team can still merge code and still cannot answer a simple "what did the provider do." That is not an IT inconvenience. That is an ownership hole in the architecture.

The problem

The apparent problem is bus factor as a slogan. We should document more.

The real problem is load-bearing folklore. The organisation runs on steps that are not in the application: who to ask, which flag, which queue to retry, which customer is special. Folklore is fast for the person who owns it and impossible for everyone else.

If you "fix" this with more meetings, you have moved the folklore into a calendar. If you fix it by writing novels, you have moved it into a wiki nobody opens during an incident.

The problem is also incentive. The expert is fast. The organisation likes fast. Every hour they spend making someone else able to be fast looks like a loss on a board that only counts tickets. Holiday-readiness will not win that board unless you name it as work. If you do not name it, you will keep being complimented for answering Slack from a train.

The tempting solution

Ask the expert to write a README this quarter. Pair once. Say we have a culture of sharing.

Or rotate people randomly onto every area so nobody is deep. You have replaced a single point of failure with a team of tourists.

Why that is not enough

Documentation that describes screens and not decisions goes stale on the next feature. People need "why we do not run this migration at noon," not a screenshot of Horizon.

Pairing that does not change ownership leaves the expert as the reviewer of record forever. The other person was a visitor.

Heroics hide the metric. If you always answer, the team never feels the gap. The gap is the product. You are papering it.

Review-as-bottleneck is the polite version. The expert does not mean to be a gate. They are the only person who has been burned by the path, so every similar change waits. Waiting looks like quality. It is also a queue with one consumer. Queues with one consumer do not take holidays. They take laptops to the beach.

Options

Keep the expert model and pay them well. Advantage: speed in the known areas. Disadvantage: hiring, holidays, and illness become incidents. This is how many small teams actually run. Naming it is at least honest.

Make every area a committee. Advantage: no single owner. Disadvantage: no single owner.

Named owners with a documented deputy and a written decision log for the sharp edges. Advantage: someone is accountable, someone can cover, the "why" is findable. Disadvantage: you have to refuse some work so the owner can write the dull page. I want this.

Automate the folklore into the product: runbooks as commands, guarded deploys, replay tools. Advantage: the system carries the knowledge. Disadvantage: you must maintain the tool. Worth it for the weekly rituals. Not for every paragraph.

Record decisions, not procedures. A page that says "we do not retry this vendor, here is why, here is the human path" will outlive a page that says which button to click in last year's Horizon. Procedures rot. Decisions stay load-bearing. Teach the decision and the current command. Skip the screenshot novel.

Trade-offs

You trade the expert's flow for the team's ability to move without them. They will feel slower. They are buying optionality.

You trade some depth. A deputy will not know every scar. They need the scars that are still load-bearing.

You should not trade away the expert's craft by making them a full-time writer. The job is to make the dangerous ten percent explainable, not to narrate every helper. If the expert disappears into a wiki, you have moved the bottleneck from production to prose.

Decision

I treat holiday-readiness as a quality of the system, not as a personality trait of being "supportive." Before I take a week, I want a deputy who has actually performed the risky path once, and a page that states the decisions that still surprise people.

If I cannot leave, that is a backlog item with a name. It is not a compliment.

I also watch my own reviews. If every risky pull request waits for me, I have built a queue that looks like quality and behaves like a single point of failure. The fix is not to review less carefully. It is to encode the scar in a test or a checklist so someone else can merge the next similar change without waiting for my mood.

Implementation / Thinking process

List the paths that would page someone: payments stuck, mail backlog, a failed deploy, a tenant mix-up, a vendor token. For each, write the decision, the danger, the command, the "do not." Keep it short. If it cannot be short, the path is too magical and should be simplified.

Change review so the expert is not the only merge. That may mean the expert writes the test that encodes the scar, then someone else ships the next change against that test. The test is the deputy that never sleeps.

When I am the expert, I practice answering with a link and a walkthrough, not a silent fix. A silent fix teaches the team to bring me live incidents as a service.

Shadow on-call is useful. So is the opposite: the expert is not allowed to touch the first hour unless the deputy asks. That sounds harsh. It is how you discover the missing sentence in the runbook.

Vendor access gets the same treatment. A shared, audited login or a second named owner is not bureaucracy. It is how you stop a password manager entry from being a production dependency. If only one human can open the dashboard, you do not have an integration. You have a priesthood.

I also ask people to teach backwards. The expert writes the page. The deputy performs the path. Then the deputy edits the page. If the page cannot survive that edit, it was a transcript, not a tool.

I keep the list of "only I know this" short enough to review in a monthly half hour. If the list grows, that is the backlog, not a personality quiz. The items are usually access, a weird clock, and a module nobody wants to open. Those three are enough to stall a week. They are also enough to plan.

Failure modes

The deputy who is named and has never been allowed to fail. Ownership theatre.

The document that is a transcript of a call. Nobody can scan it when they are scared.

The expert who documents as revenge — so much detail that the page is a second codebase. Brevity is a kindness.

The rotation that puts a new person on mail week without a floor. Sharing is not the same as throwing.

The holiday that is "covered" because someone has the password and not the decision. Access without the "do not" list is how you get a brave deputy and a worse week. Coverage is the path performed once, not the vault opened.

Operational consequences

When this works, the expert gets to do deeper work instead of being a router. When it does not, they burn out and the organisation is surprised, which is an odd reaction to a single point of failure doing what single points do.

Stakeholders will still try to "just ask you." You have to redirect in front of them, or you have taught them the old design.

There is a hiring consequence too. If the only way to be useful is to have been present for last year's scars, you cannot onboard. New people will look slow. They are not slow. They are locked out of a language that was never written down. You will then conclude that "good people are hard to find" when you have made the job a memory test.

Lessons

Leverage on a small team is the number of correct actions someone else can take without you. Code quality is part of that. So is the unglamorous page that says why the job is not on the default queue.

If you are proud of being the only one who can do it, you have optimised for identity. The product needs a system. Identity does not page well at 2am, and it does not board a flight. A system does both, badly at first, then better.

What I would do differently today

I would have named deputies before we had a painful holiday, not after. Pain is a poor planning tool. A calendar is a better planning tool.

I would have written decision records for the weird clocks — the 02:10 job, the vendor who must not be retried, the table you do not vacuum at noon — before I wrote more onboarding prose. New people do not fail on how to run npm. They fail on the weird clocks.

I would have treated "I will just do it, it is faster" as a debt with an owner. Sometimes it is the right hour. It is never a strategy. If the same silent fix happens twice, the third time is a page or a command, not another hero evening.

Closing thought

A team that needs you on holiday does not have a system. It has a bottleneck with a passport. The passport should be allowed to leave. The work of leadership is to make the dangerous knowledge findable, the ownership shared enough to survive a week, and the silent heroics slightly embarrassing.

Context

The systems are the usual mix: Laravel, MySQL, Redis, queues, a payment or mail vendor, a mobile client. Logs exist. They are never the whole story. Metrics exist. They are usually about machines. The user is talking about a life: I paid, I did not get the letter, the list is wrong, it worked for my colleague.

You will not get a perfect dump. You will get a time, a person, a screen, and a feeling. That is enough to start if you treat it as the first layer of a timeline, not as a diagnosis.

I keep a short private list of questions that have wasted my time: which environment, which tenant, which binary if it is mobile, which job id, which provider id. If the ticket has none of those, the first engineering act is to ask, not to open the code. Opening the code feels like work. It is often a way to avoid the timeline.

The problem

The apparent problem is the symptom. It is slow. It is wrong. It failed.

The real problem is missing causality. You have effects and you do not have the chain. Local reproduction is a luxury that requires the chain to be portable. Many operational bugs are not portable. They are situated: this tenant, this job, this deploy, this payload.

If you skip reconstruction, you will "fix" a nearby thing. Nearby things are plentiful.

Production also withholds the replay button. You cannot always re-enter the same tenant, the same payload, the same vendor mood. So the work is not "make it happen here." The work is "make a chain that is good enough that changing the wrong thing would be embarrassing." If you cannot point at a row, a job, a deploy, or a vendor event, you are not debugging. You are renovating.

The tempting solution

Reproduce it locally, or keep changing production until the ticket goes quiet. Add logging after the fact in a panic, without knowing which question the new line answers.

Blame the vendor, the cache, the last deploy, the user. Those are hypotheses. They feel like conclusions because they are available.

Why that is not enough

Local environments lie by being clean. They have one tenant, one worker, yesterday's data, and your patience.

Restarting things destroys evidence. The queue you flushed may have been the only copy of the payload. The box you cycled may have been the only place the file landed.

A single log line without a request id, a tenant, a user, and a clock you trust is a postcard from nowhere.

Metrics that say CPU was high do not say which business action was in flight. You will scale the wrong thing. I have already written about that temptation with databases. It shows up here as a personality: when in doubt, reboot the graph.

Incomplete evidence is not an excuse to stop. It is a reason to be precise about what you cannot see. "We have no job id" is a finding. "We have no vendor event" is a finding. Those findings tell you whether the next step is a code change or an instrumentation change. Mixing them up is how you ship a patch that cannot be proven.

Options

Guess and ship a patch. Advantage: you look fast. Disadvantage: you will ship the patch next to the real bug.

Wait for it to happen again with more logs. Advantage: sometimes correct. Disadvantage: you are using users as probes.

Build a timeline first: user action, application logs, job logs, database row versions or updated_at, vendor dashboard, deploy history, feature flags. Then pick the next measurement that would kill the largest remaining hypothesis. Advantage: you spend time on questions. Disadvantage: it feels slow to people who want a restart. I want this to be the default.

A full observability platform before you have questions. Advantage: pretty. Disadvantage: you will still not look at it if the culture is restarts. Buy tools for the questions you already know you cannot answer, not as a substitute for a timeline.

Copy the record into a scratch environment and still fail to reproduce. Advantage: you tried. Disadvantage: you may have copied the row and not the job, the flag, or the vendor payload. Reproduction that ignores situated state is a second laptop lie. I would rather a timeline in production evidence than a staging clone that is missing the weather.

Trade-offs

You trade the comfort of "doing something" for the discomfort of not touching the box until you know what you would destroy.

You trade some user time. A careful investigation can be longer than a lucky restart. A lucky restart that erases the trail can be much longer.

You should not trade away a written timeline because it feels formal. Three bullets in the ticket are enough. Memory is a bad shared document. The next engineer cannot inherit your certainty. They can inherit your clocks.

Decision

I start with time. When did they see it. What deploy was live. What else happened in that window — a job backlog, a vendor status, a migration. Then I ask what evidence would have to exist if the popular hypothesis were true. Then I look for that evidence, or I note that we cannot see it, which is itself a finding.

I do not start with a change unless the system is on fire in the ordinary sense: we are losing money, leaking data, or cannot log in. Even then, the change is to stop the bleeding, and the timeline still gets written.

I also separate "we mitigated" from "we understood." Restarting a worker can be a mitigation. It is not a cause. If the ticket closes at mitigation, the next Tuesday will reopen it with less evidence. Write the residual risk in the ticket: we stopped the bleeding, we still do not know why the job stalled, here is the next measurement.

Implementation / Thinking process

Give every request and job a correlation id and put it on the logs you already have. This is dull and it is how reconstruction becomes possible next time.

Prefer facts that have clocks: row timestamps, queue available_at, provider event times. Align them. Clock skew is a character in this story.

Read the row. I still see people debug from the UI and never look at the record. The UI is a rendering. The record is closer to the crime.

Ask what the user could not see. Pending jobs, dead letters, a payment in captured-not-allocated. Those are the rooms the screen does not mention.

When you add a log, write the question above it in the pull request. If you cannot, you are hoarding string.

Compare clocks before you compare feelings. "It failed after deploy" is a hypothesis that needs the deploy time and the first bad event. I have watched people blame a release that landed after the first failure. The timeline would have saved the argument.

If the path crosses a vendor, open their evidence as a first-class source, not as a last resort. Their event id belongs on your timeline. If you cannot find it, that absence is a finding about your integration, which I have written about elsewhere as a neighbour problem.

I write hypotheses as sentences that can die. "The job never ran" dies if we find the job id. "The vendor never saw it" dies if their dashboard has the event. "The user double-submitted" dies if there is one intent. A hypothesis that cannot die is a prejudice. Prejudices ship patches.

Failure modes

The working hypothesis that you fall in love with. Disconfirm it on purpose. Ask what would prove you wrong.

The "cannot reproduce" close. That is a status of your laptop, not of the system.

The extra logging that includes a payload you should not store. Reconstruction does not outrank privacy.

The incident that becomes a personality story about a colleague. If your timeline needs a villain, it is not finished.

The patch that "might help" and closes the ticket. If you cannot say what evidence would have changed if the patch were right, you have shipped a ritual. Rituals accumulate. Evidence does not.

Operational consequences

Teams that reconstruct get calmer. They ask for ids instead of feelings. Support learns to send the time and the reference. That is a cheaper culture change than a new APM bill, and it makes the APM useful later.

Teams that reboot as a reflex train production to only fail in ways that survive a reboot. Those ways are nastier.

You will also change how tickets arrive. Once you ask for time, tenant, and a reference, people start sending them. That is not bureaucracy. That is how a timeline gets a first layer without an engineer excavating chat. If you never ask, you will keep receiving novels and keep opening the code to feel busy.

Lessons

Debugging without reproduction is historical work. You are not a mystic. You are trying to make the next observation more informative than the last.

If you cannot tell the story of the action, you are not ready to change the code. You are ready to change what you can see. A patch without a story is how a codebase collects souvenirs from weeks nobody can reconstruct. Souvenirs compile. They do not explain Tuesday.

What I would do differently today

I would have put correlation ids on jobs the same week we put them on HTTP. I treated background work as secondary. The bugs did not.

I would have written the timeline in the ticket even when I was sure. Sure is when you skip steps. The steps are for the sure days.

I would have treated "cannot reproduce" as a prompt to improve evidence, not as a close reason. If we cannot see it, the next commit is often a better clock or a better id, not a speculative if. Speculative ifs are how codebases fill with scars that never matched the crime.

Closing thought

A bug you cannot copy is still a fact in the world. Treat it like one. The laptop is optional. The clocks are not. Build the chain from the clocks you have. The restart will still be available when you know what you are restarting, and why, and what evidence you are about to delete.

Context

The pattern I know is a Laravel API, a web admin or portal, and a Flutter client that talks HTTP, stores a token, and sometimes receives a push. Networks are hostile. Users do not update. Review cycles are slower than your patience.

I work with mobile engineers rather than pretending to be one. My job on this boundary is the contract: what the phone may assume next month, and what we will do when it assumes last month.

That contract includes the ugly verbs. Login. Refresh. Upload. Pay. Resume a list. Recover from a 401 in a tunnel. If we only design the happy GET, we will meet the ugly verbs as crashes, and we will blame Flutter. Flutter was obeying physics.

The problem

The apparent problem is feature parity. The app should do what the website does.

The real problem is time-to-force-upgrade is not a number you own. Until you are willing to block old binaries, every additive backend change must be survivable by a client that has never heard of it, and every breaking change is a coordinated programme.

Offline and slowness make it worse. The phone will retry. The phone will show cached lists. The phone will submit twice. Your pretty POST is now a distributed system with a human in the loop.

The problem is also attention. A website user who hits an error can refresh. A phone user on a platform may not. They will tap again, background the app, or decide the institution is broken. Your API's manners — idempotency, stable errors, resume — are the product those people meet. The Flutter code is how that meeting is dressed.

The tempting solution

Change the API, update the app, submit to the store, done. This works in a slide. In a week you have a mixture of binaries, some still on a beach with a week-old build.

Or build a private, mobile-only API that you break freely because "only we consume it." You still do not consume it on your schedule. You consume it on the store's.

Why that is not enough

Auth sessions that assume a short-lived browser tab will punish a phone. Tokens need refresh and revocation. A logged-in phone is a long-lived credential in a hostile pocket.

Push is not a guaranteed inbox. It is a hint. If the product requires the push to have been seen, you have designed a wish.

Pagination and search that were designed for fat web connections become jank and duplicate rows. The contract needs to say how to resume.

Error payloads that were designed for a page reload become uncaught exceptions. Mobile clients need stable codes, not only English.

Web and mobile also disagree about identity of a session. A browser cookie that dies with a tab is a different animal from a token that lives in secure storage for months. If you design auth for the tab and issue it to the pocket, you have given a long-lived credential the manners of a short one — or the reverse: you have made the website carry a phone's lifetime. Pick the stricter story and implement it twice, or you will implement the sloppy one twice.

Options

One public API for web and mobile, additive and strict about compatibility. Advantage: one test suite, one model. Disadvantage: the slowest client sets the pace. That is often correct.

A backend-for-frontend for mobile. Advantage: you can shape payloads for radio and battery. Disadvantage: two surfaces to break. I add this when the mobile needs aggregates the web should not drag around, not as a way to hide sloppy versioning.

Force upgrade. Advantage: you get to break things. Disadvantage: you spend goodwill, and you still need a window. Force upgrade is a tool for unsafe old clients, not a substitute for manners.

Accept that some features are web-only until the app catches up. Advantage: honesty. Disadvantage: product people dislike forked capability. I still prefer a fork in capability to a fork in truth.

Ship the backend change in two releases: first the additive tolerance, then the app that uses it, then — much later — any floor. Advantage: the wild binaries have a chance. Disadvantage: you live with dual shapes. That is the cost of a store. Anyone who wants one release for both clocks is asking the pocket to attend the stand-up.

Trade-offs

You carry fields you are ashamed of. You carry query parameters you renamed in your head but not in the world. That shame is cheaper than a store emergency.

You accept that "released" for mobile means "available," not "adopted." Metrics of binary versions, if you have them, should humble your cleanup plans. If you do not have them, assume a long tail.

You should not accept silent data loss because the phone was offline. Queue the write or refuse it. Do not pretend it landed. A 200 you never sent is not optimism. It is a second ledger.

Decision

I treat the mobile client as a customer of the API with a slower clock than the website. Additive changes land first. The app uses them when it can. Breaking changes wait for a version floor we can defend.

Idempotency keys on creates that the user can tap twice. Refresh tokens that we can kill. Payloads that degrade: unknown fields ignored, missing optional fields tolerated. That is not sloppy. That is adult.

I also want a kill switch that is not "hope they update." A config flag the app already knows how to read can disable a broken flow without a store review. That is not a substitute for compatibility. It is how you survive the week between discovering a bad payload and getting a binary adopted.

Implementation / Thinking process

Contract tests include an old fixture. If the new code cannot read what last quarter's app sent, we have already failed.

List endpoints that feed infinite scroll need a cursor or a very honest snapshot. "Page 2" after an insert is how users see ghosts and duplicates.

File uploads and poor radio need resume or a patient failure, not a 30-second request that dies in a tunnel.

When we add a field the old app must not require, we do not add it as required in the general schema and then "make the website send it." The next client will be a script that does not.

Push sends a pointer, not the whole privileged payload. The app fetches through the same auth as everyone else. Side-channel data in a notification is how you leak.

Auth refresh has to be designed as a state machine the app can run in the background. If every 401 dumps the user to a login screen, you will train them to hate the product, and you will also hide the real failure: the refresh token died, the clock is wrong, or the server started requiring a scope the old binary does not request. Those are different repairs.

When we change pagination, we keep the old query working. Mobile lists are long-lived in memory. A new cursor scheme that 400s the old page parameter is a crash on a train.

I also want the backend to tolerate an app that is briefly a liar. It will show a cached list. It will retry a POST. It will keep a draft the server never saw. Your job is not to punish that. Your job is to make the lie reconcilable: the list can refresh, the POST can be the same intent, the draft can fail with a sentence. A backend that assumes a fresh browser on every tap will call ordinary mobile behaviour abuse.

Failure modes

A required field added for a new web form. Old app cannot post. Users look incompetent. They are not.

A datetime format that changed from date to datetime. Parsers are petty. So are stores.

Assuming the app will refresh its config on next launch. Some users do not launch. Some launch offline. Config that is safety-critical needs a default that fails closed.

A single global error toast for every 422. The user cannot fix "eligibility." They can fix "phone number." If you do not distinguish, they tap again.

A push that contains the privileged fact because the fetch seemed expensive. The notification shade is not your ACL. Fetch through the same door as the rest of the product, or you have built a side channel with a bell on it.

Operational consequences

Support will report "the app" when the API changed. You need a way to ask which version. If you cannot, you will guess, and you will guess the latest.

Release planning now has two clocks. If product ignores the store clock, engineering will become the people who "block" features. Name the clock in the planning meeting. It is not drama. It is physics.

Support will also meet offline as a personality. "They said they submitted." Maybe they did, on a radio that died before the 200. If you cannot show the intent as pending or failed, you will argue with a human about a packet. The argument is the cost of a create that had no memory.

Lessons

A backend that serves a phone is a compatibility product. The website is a privileged client that happens to deploy with you. Do not let it set the manners for everyone else.

If you cannot force an upgrade, you have not finished the change. You have published a hope. Hope is not a release plan for a binary you cannot recall.

What I would do differently today

I would have versioned the mobile contract in conversation before we had a second screen. The first screen teaches bad freedom.

I would have added idempotency on the first tap-happy create, not after support saw doubles. Phones double-submit. That is not a user error. That is a transport.

I would have put a minimum-version signal in the first payload the app already fetches, even if we never blocked anyone. Building the pipe is cheap when you do not need it. Building it during a bad field is how you ship a panic binary.

Closing thought

The phone will keep the contract you already broke. Design the API for the binary you cannot recall. Deploy the website as often as you like. The pocket is on another calendar, and it does not attend your stand-up.

Context

I lead delivery on a small product group: web, APIs, a Flutter client, the operational systems underneath a college. The work is a mix of features, integrations, and the kind of maintenance that never looks like a launch. There is no separate architecture office. There is no project manager whose only job is to protect a sequence. If a decision about risk is not taken inside the team, it is taken by calendar pressure.

That shape is common and it is specific. In a large org, some of these decisions live in staff-plus roles or in written programmes. In a small team they live in the same person who still opens a pull request. The danger is mixing those modes without noticing. You think you are being helpful in the code. You are actually leaving the decision layer empty.

Stakeholders are close. That is good for truth and bad for impulse. A request can go from hallway to ticket without passing through a design that anyone would defend a month later. Someone has to be the place that impulse stops and becomes a choice.

The real problem

The apparent problem is throughput. People want to know why a thing is late, or why two engineers solved the same category of problem in two styles, or why a "simple" integration keeps returning.

The deeper problem is unowned trade-offs. Every product has more good ideas than it has careful hours. If nobody decides what we will not do, the backlog decides by overflow. If nobody decides which boundary is real, every feature punches a new hole in the monolith. If nobody decides what "done" means for a payment or a migration, done means shipped-enough-to-demo.

A second deeper problem is false delegation. You can ask the team to "own quality" and still be the only person who is allowed to say the date is wrong. That is not empowerment. That is leaving people responsible for outcomes they cannot steer. Real delegation includes the authority to change the work, not only the duty to finish it.

I also had to admit a personal version of the problem. I like being the person who can still do the task. It is visible. A decision about scope is invisible until it saves you, and by then nobody thanks it. The ego wants the visible thing. The role needs the other.

Constraints

Time is not abstract. Term dates, payment windows, and external vendors create real clocks. A lead who ignores those clocks is not principled. They are decorative.

The team is mixed in seniority. Some people can be given a vague problem. Some people will produce a careful mess if you give them a vague problem. Delegation has to match the person, which means the lead still designs the size of the box.

I still write code. If I disappear into "alignment," I lose the right to smell a bad shortcut. If I write too much of the critical path, I become the bottleneck I was hired to remove. That tension does not resolve. It is managed week by week.

Stakeholders can overrule. A tech lead does not win every argument. What they can do is make the cost legible. If the organisation still chooses the expensive path, that is a recorded decision, not a silent one. Recording it is part of the job.

And we cannot pause the systems. Decisions happen while production is live. That is why "we will decide the architecture after this release" is usually a way of deciding to keep the current one.

Options considered

Lead as the best pair of hands. Advantage: short-term delivery, credibility, comfort. Disadvantage: the team's ceiling becomes your calendar. Risk: you train people to wait. Maintenance of the organisation is worse even if the repository looks loved. The business gets a hero and a queue.

Lead as a process clerk. Stand-ups, tickets, status. Advantage: visibility. Disadvantage: nobody is deciding the technical bets, so process becomes a way of watching accidents arrive on time. Risk: you become the person who knows the dates and not the person who knows the blast radius.

Lead by consensus only. Advantage: buy-in, fewer lonely calls. Disadvantage: consensus is slow, and some decisions are about saying no to a likeable idea. Risk: the loudest or most senior-sounding engineer wins, which may not be the one who will live with the on-call. I want debate. I do not want a vote on every boundary.

Name a small decision set and keep it visible. What we will not build. What debt we accept, and until when. Which module owns a fact. Whether a date is a wish. Whether an integration is allowed to touch money before it has a replay story. Advantage: people know where to escalate. Disadvantage: you will be the person who says no. Risk: you hoard decisions that should have been pushed down. The work is to keep the set small on purpose.

Decision

I treat five calls as mine unless I have explicitly handed one to a named person with the authority to finish it.

What we are not doing this cycle, even if it is good. What quality of debt we are accepting, written down, with a trigger to revisit. Where a domain fact is allowed to be written. Whether a committed date is still honest. Whether a change is allowed to cross a risky seam — payments, tenancy, identity, the jobs that talk to the outside world — without extra eyes.

I chose that list because those are the decisions that, if fumbled, create work for everyone else. A missed naming convention is annoying. An unowned payment state is a month of finance. A date that was never challenged is how you get a team that stops telling you the truth.

Why not more? Because a lead who decides the implementation of every ticket is not leading. They are hiding in taste. The team should own how a screen is built once the seam and the risk are clear. If I cannot let that go, I do not have a team. I have assistants.

Implementation / Process

The process is lighter than the words suggest. Intake gets a question that sounds rude and is not: what happens if we do not do this. If the answer is thin, the work is a preference. Preferences can wait.

Risky seams have a written owner and a written "done." For money, done includes a state you can explain and a replay. For a migration, done includes a rollback that is not theoretical. For a mobile API change, done includes what old clients will see. I do not need a long RFC for every ticket. I need the done that matches the blast radius.

Dates get interrogated in private before they get defended in public. If I have already sold a date I do not believe, I have chosen politics over the team. Sometimes I still have to carry a date I dislike. Then the job is to cut scope in the open, not to hope.

I stay in the code on purpose, in review and in the paths I do not want to learn about from an incident. I do not take every interesting ticket. That was a harder habit than it sounds. Interesting tickets are how you keep feeling like yourself.

When I delegate a decision, I say the words. "You own whether we extract this job or leave it in the monolith. I will live with the call." If I then override it casually, I have taught the team that ownership is theatre.

Problems and failures

I have overridden a call after saying I would not, because I got scared in review. The code may have been better. The team was worse. If you need to take a decision back, take it back as a recorded change, not as a drive-by comment.

I have also under-decided and called it trust. A vague "see what you think" to someone who needed a box produced a thoughtful design we could not operate. That was my failure to size the work, not their failure to be senior.

Saying no without an alternative made me look like a gate. The useful no includes the smaller yes, or the later date, or the risk we are accepting if they push anyway. Stakeholders can handle a trade-off. They cannot handle a mood.

I stayed too close to one technical area because I liked it, and too far from another because I found it dull. Dull areas — access, naming of statuses, the admin screen finance actually uses — are often where the organisation lives. Neglect is a decision.

There were weeks when I confused being busy with having decided things. A full calendar can hide an empty decision log. The team feels that as drift.

Trade-offs

I trade some personal coding joy for fewer surprises. I still write. I do not write the most of the fun path. That will always itch a little. It should. If it stops itching, I have probably drifted too far from the work.

I trade speed of agreement for clarity of disagreement. Writing down a no slows the meeting. It speeds the next month.

I do not trade away accountability when I delegate. If I handed you the call, I still stand next to the result in the stakeholder conversation. Otherwise people will not take the next call.

Result

The team became less surprised by me and more willing to surprise me with a finished decision. That is the direction I want. Planning arguments started happening before the code, which is cheaper even when they are uncomfortable.

I also became easier to replace for a week. That is an unromantic success criterion and a good one. If nothing can move because I have the only copy of the no, I have built a fragile organisation around my taste.

Work did not become calm. It became easier to locate. When something went badly, we could usually point to a call we had made, not only to a ticket we had rushed.

What I would do differently today

I would have written the decision set down in the first month, badly, instead of waiting until I could make it sound wise. A crude list would have saved a few silent accidents.

I would have practised the public no earlier on low-stakes work, so the high-stakes no did not become the first time people saw me do it. Muscle memory belongs in leadership too.

I would spend even less energy on being the person who can still do every task, and more on being the person who notices when a date has become a lie. The lie is more expensive than an unclaimed bug.

Broader lesson

A tech lead is not a title for the strongest engineer. It is a role that absorbs a few decisions so the rest of the team can move without waiting for telepathy.

If you cannot name the decisions you refuse to leave ownerless, the calendar will name them for you. The calendar is a terrible architect.

Closing thought

Stay close enough to the code to keep your nose. Stay far enough from the tickets to keep your hands off the decisions that only work when they have a name. The team does not need you to be everywhere. They need you to be findable when the trade-off is real.

Context

The work I care about here is PHP applications on familiar hosting — Forge, AWS, Docker — with GitHub Actions or a similar runner, and databases that cannot be treated like cattle. You can replace a container. You cannot casually replace a term of invoices.

Releases also include the unglamorous: migrate, restart workers, clear config, warm something, watch something. If the pipeline only runs tests, you have continuous integration and a folklore production.

I have used GitHub Actions, Bitbucket pipelines, and host-side deploy hooks. The brand does not matter. What matters is whether the same commit produces the same artefact, and whether a human can point at that artefact in production. If those two sentences are soft, you will debug "what is live" as if it were a mystery novel.

The problem

The apparent problem is toil. Deploys are manual. Let us automate them.

The real problem is surprise. Changing production is how you introduce new failure. The organisation wants features and also wants Tuesday to resemble Monday. Those wants fight. Delivery is the process that makes the fight boring.

If your pipeline cannot talk about rollback, migrations, secrets, and environment drift, it is not managing risk. It is typing for you.

The other half of surprise is people. A release that is technically fine can still be a bad Tuesday if it lands in the middle of enrolment, a payment window, or a mail run. Frequency is not automatically virtue. Frequency is virtue when the blast radius is small and the undo is real. A pipeline that can ship twelve times a day and cannot reverse a migration is a machine for making irreversible afternoons cheaper to start.

The tempting solution

On push, install, test, deploy. One workflow. Main is production. Feel like a grown-up.

Add more checks when something burns you, until the pipeline is a museum of past pain and a twenty-minute wait. People start merging because they are tired, not because they are confident.

Why that is not enough

Tests that do not include the dangerous paths — money, tenancy, auth, jobs — produce green that is not about production. I have more patience for a short suite that covers those than a long suite that covers helpers.

Database migrations do not roll back because the workflow has a rollback job. Expanding a column is not the same as contracting it. If you do not have expand/contract, your "undo" is a restore conversation.

Environments lie. Staging that does not run the workers, or does not have the same config shape, will bless deploys that fail in the only environment that matters.

Secrets that live in the runner and in the server and in a chat will eventually disagree. The pipeline cannot save you from three truths.

Feature flags are not a substitute for a broken migration. They are a way to decouple release from launch when the code can tolerate both versions.

There is also the artefact problem. If CI installs dependencies on the runner and production installs them again on the box, you tested a cousin of what you shipped. "It passed" then means "a similar looking tree passed." I want one build. Promote that build. If you cannot, say so, and stop using the green check as a synonym for "what is live."

Options

Manual deploy from a laptop. Advantage: someone is paying attention. Disadvantage: the laptop is the process. Holidays become architecture.

Fully automatic deploy on every green main. Advantage: small diffs. Disadvantage: a flaky test or a compromised token is a production event. I like small diffs. I like a human on the last step when the blast radius is money or mail.

Build once, promote the same artefact, separate the decision to release. Advantage: you know what you tested. Disadvantage: more plumbing. This is the shape I want as soon as "it passed on CI" and "what is on the box" can diverge.

Trunk plus flags versus long-lived release branches. Branches feel safe and rot. I prefer trunk and honesty about flags over a branch that only exists because we are afraid.

Trade-offs

You trade speed of merging for speed of believing. A pipeline that is fast and flaky trains people to retry. A pipeline that is slow and true trains people to batch. I would rather fix flakes than add retries.

You trade some autonomy. A required check is a gate. Gates need owners. An unowned required check becomes a local legend about who knows the YAML.

You should not trade away a dry run for migrations in production-shaped data, even if it is a subset. The first time you meet a trigger or a long lock should not be the afternoon. A pipeline that never saw the real shape of the table is a costume rehearsal.

Decision

I want the pipeline to answer three questions: is this artefact fit, are we choosing to release it, and can we undo the parts that are undoable. Fitness is tests plus the boring static checks. Choice is a separate step for anything that talks to students or money. Undo is documented, not implied.

Workers are part of the release. A web node on new code and a worker on old code is a split brain I have already met in mail. The pipeline should make that split hard.

I also want the release to say what it will not do. A pipeline that deploys code and "someone will run the migration" is two processes pretending to be one. Either the migration is in the path, with a lock and a log, or the release is incomplete and should be named that way.

Implementation / Thinking process

Keep install and test deterministic. Pin versions. Cache what you understand. A build that depends on the internet's mood is not CI.

Do not hide deploy in the same job as test without a name. "CI passed" and "production changed" should be different sentences in the log.

Migrations run with a plan: expand first, deploy code that can live with both shapes, contract later. If someone needs a flag-day column rename, that is a product pause, not a clever YAML.

Secrets come from one store the server and the runner agree on. If a human can deploy with a .env they edited by hand, you have a second pipeline.

After release, look at the paths that would embarrass you: login, a payment return, a queued send. That is not a full QA theatre. It is a refusal to walk away from the room.

Feature flags belong in this path when the risk is behavioural, not structural. A flag can hide a screen. It cannot hide a destructive migration. Mixing those two ideas is how teams "roll back" a flag and still live with a column that cannot go back.

Environment parity is a gradient, not a religion. I do not need staging to send real mail. I do need it to run the same job names, the same config keys, and the same migrate command. If the keys differ, you will discover a missing secret in the only place it hurts.

I want the release notes to mention data. "Code deployed" is incomplete if a migrate ran, a flag flipped, or a worker binary changed. Those are different undo stories. If the chat only says shipped, you will attempt the wrong undo. The pipeline can print the list. A human can still press the button. The list is the product.

Failure modes

The skipped check "just this once." There is no once. There is a new default.

A rollback that rolls the code and not the data, or the data and not the workers.

A staging that uses a shared database with production names. You will migrate the wrong place. I will not invent a story. I will say the design invites it.

A pipeline so privileged it can do anything, owned by no one, with a token in a fork pull request. You have automated a breach.

The green check that required skipping a flaky test. You did not buy speed. You bought a new meaning for green. The next person will skip a real one and cite your precedent.

Operational consequences

When delivery is trusted, people ship smaller things. That is the cultural prize. When it is not, people batch "to be safe" and invent the risky deploy they were trying to avoid.

Non-engineers will start asking "is it on production." If you cannot answer without SSH, the pipeline is not yet a product.

You will also discover that failed deploys are a product of their own. A half-applied migrate, a worker that did not restart, a config cache that still has last week's secret name — those are not "the pipeline being flaky." They are states. If the only recovery is "run it again," you will run it again against a world that is already half new. Name the failed states in the runbook before you need them at 18:40.

Lessons

CI/CD is not a badge. It is a way of making change to a live system less theatrical. The artefact, the decision, and the undo are the plot. YAML is scenery.

If the team does not believe the green check, you do not have automation. You have decoration. Belief is the SLO of a pipeline. Without it, people invent a second, unofficial release path, and that path is the one that will hurt you.

What I would do differently today

I would have split test and release on the first production app, not after the first confusing rollback. Combined workflows hide the decision.

I would have treated worker restarts as a first-class step instead of a footnote in a README. The README is not on call.

I would have written the expand/contract rule on the first column rename, not after the first failed undo. A rename that needs downtime is a release type. If the pipeline cannot say "this is a flag day," people will treat it like a Thursday deploy and be surprised when Thursday cannot go backwards.

Closing thought

A safe release is one you can describe: what we proved, what we shipped, what we can still undo, and who we will surprise. If your pipeline cannot say those sentences, it is typing. Typing is useful. It is not the same as being allowed to change the week.

Context

Laravel applications accumulate Redis the way they accumulate helpers. Cache, sessions, queues, rate limits, a lock around a webhook. One instance, several jobs. That consolidation is tidy until the jobs disagree about how important durability is.

The screens that attract cache are the obvious ones: dashboards, navigation counts, catalogue-like lists, permission-heavy menus. In a college admin, that might be an inbox count. In a multi-tenant storefront, a category tree. The user-facing win is real. So is the ticket that says the number is wrong.

I have also seen Redis used as a meeting-avoidance tool. The query was ugly. Nobody wanted to own a read model. A key felt cheaper than a conversation. Six months later the key is in four writers and nobody can flush it without a ceremony. The conversation still happens. It is just worse.

The problem

The apparent problem is latency. The database is doing the same work too often.

The real problem is whether the answer is allowed to be old, and for whom. A public marketing page can be old. A payment balance cannot. A permission bit cannot. A tenant-scoped list that forgot the tenant in the key cannot.

If you cannot say the allowed staleness, you are not choosing a TTL. You are gambling.

There is a second real problem: ownership of the copy. The database has an owner — a migration, a model, a team. A cache key is often owned by whoever was last embarrassed by a slow page. That person leaves. The key remains. The next writer does not know the key exists. You now have an unowned second store. Unowned stores do not invalidate. They haunt.

The tempting solution

Cache::remember with a round number. Five minutes. One hour. Forever, plus a hope that you will forget the key on write. Hope is not invalidation.

Or cache the entire graph of a page because the page is slow. Now every write in the graph is a reason to flush a lake, so you flush nothing, and the lake goes green.

Why that is not enough

Invalidation is the product. Reads are the easy half. Writes happen in more places than the controller you remembered: jobs, admin, imports, another app. If any writer is outside the forget() call, you have a lie with a longer TTL than your attention.

Stampedes happen when a hot key expires and every request rebuilds it. You wanted less load. You created a cliff.

Redis as a second source of truth is how you get split brain. Session in Redis, user in MySQL, a permission cached between them. A change in one place is a ghost in another.

Locks and rate limits deserve more respect than "we already have Redis." A cache that evicts under memory pressure will drop your lock or your limiter if you mixed them with disposable keys. Eviction is not a policy you want for a lock.

There is also the write path people forget: "we will delete the key in the observer." Observers are easy to skip. A raw update, a migration, an artisan command, an import, a second app on the same tables — none of them will fire your observer unless you made them. Invalidation that lives in one happy controller is a wish about how writes will happen.

Options

Do not cache. Fix the query or take the report off the request. Advantage: one truth. Disadvantage: you may still be slow. Often the right first option.

Cache with a TTL only, no write invalidation, on data that may be old. Advantage: simple. Disadvantage: you must be honest in the UI that the number is as of a time.

Cache with explicit invalidation on the writes you own. Advantage: fresher. Disadvantage: every new writer is a bug. This needs ownership, not a helper.

A snapshot table or a projected read model updated by the same transaction or a job. Advantage: you can index for the page. Disadvantage: you are doing the hard work people pretend Redis avoids.

Separate Redis roles. One instance for disposable cache, another for queues and locks, or at least separate prefixes and eviction policies. Advantage: a cache flush does not become an outage. Disadvantage: more things to run. I would rather have two small honest instances than one clever soup.

Trade-offs

You trade a little staleness for a lot of load, or you trade a little load for a lot of invalidation code. There is no third thing, only people who have not met the second bill yet.

You trade operational simplicity. Redis is now in your incident story. "Is Redis up" becomes as important as "is MySQL up," except Redis failures look like random wrongness rather than a clean down page.

You should not trade correctness on money, tenancy, or auth for a faster nav badge. Those facts can be slow. They cannot be approximately someone else's.

Decision

I cache when I can say, in one sentence, what is stored, who may see it, and which writes make it false. If I cannot, I do not cache. I fix the question or I accept the time.

I keep queues and locks out of the same eviction story as page fragments. That is not purity. That is not wanting a dashboard cache to evict a payment lock.

I also refuse to cache as a way of hiding N+1. If the page issues a hundred queries, a cache will hide the shame until the key expires at the worst moment. Fix the shape first. I have already made that argument about bigger databases. It applies here with a shorter TTL.

Implementation / Thinking process

Keys include tenant, role, and the meaning of the number, not just a table name. A key called users is a confession.

Prefer small payloads. A cached HTML page is a hostage situation when a footer changes. A cached count is a fact.

Invalidate next to the write, in the same module that owns the fact, not in a random observer someone will delete. If two modules write the fact, the cache is already telling you the ownership is wrong.

For stampedes, a lock around rebuild or a slightly staggered TTL is dull and effective. Clever probabilistic expiry is optional.

Measure hits if you want, but measure wrongness more. Support tickets about stale numbers are the real hit rate.

Be careful with "warm the cache on deploy." Warming can stampede just as well as expiry, and it can warm the wrong tenant if your seeder is naive. I would rather a slow first request than a clever warm that writes one customer's page into a global key.

Sessions deserve a separate sentence. Putting sessions in Redis is not the same as caching a dashboard. A session flush logs people out. A dashboard flush makes a number blink. If those share a FLUSH command in a runbook, the runbook is a weapon.

I also want a written owner per family of keys. Dashboard counts are one family. Permission bits are another. They do not share a TTL, a flush, or a "just cache it" instinct. If a family cannot name who invalidates it, it does not get a key. That rule sounds harsh until you have flushed the wrong family to fix a badge.

Failure modes

The key that forgot the tenant. I have mentioned this in tenancy. It belongs here too. Fast leakage is worse than slow leakage.

The flush-all that a developer runs in production because a key was stubborn. You have just logged everyone out or emptied a queue, depending on how sociable your Redis is.

A negative cache — "not found" — that outlives the create. New records appear to not exist. This feels haunted.

Caching authorised data under a key that is only the object id. The next user gets a faster view of someone else's inbox.

Caching a computed permission and then changing a role in the database. The person is no longer staff. The key disagrees for the rest of the TTL. That is not eventual consistency. That is a delayed "yes" on a door you already closed.

Operational consequences

Once a cache exists, people will put more in it. The first key is a gateway drug. You need a written list of what is allowed to live there. If the list is "whatever is slow," you will cache a permission check and meet it in an incident.

Incidents involving caches are social. The database is right. The screen is wrong. Engineering looks incompetent while being, in a narrow sense, fast.

The other consequence is diagnostic fog. When a number is wrong, the first question is now "is it stale or is it false." That question did not exist when there was one store. You will spend meetings on it. If you cannot answer it from a header, a timestamp on the payload, or a "as of" in the UI, you will answer it with a flush, and the flush will become the culture.

Lessons

A cache is a second database you are pretending not to operate. If you would not accept that database being wrong, do not put the fact in Redis.

Speed is not the same as freshness. Choose which one the screen is selling.

What I would do differently today

I would have separated cache from queue earlier, even when the bill looked silly. Mixed Redis is a rite of passage I do not need to repeat.

I would have required the invalidation sentence in the pull request, next to the remember() call. No sentence, no cache. TTL-only is allowed if the sentence is "this may be five minutes late and that is fine."

I would have put an "as of" on any cached number a human might argue with. A badge without a time is a dare. A badge with a time is a fact people can live with. The extra words in the UI are cheaper than the ticket that says the system is lying.

Closing thought

Redis will happily remember a lie. That is its talent. Use it when the lie is bounded and named. If the only thing you know is that the page was slow, you are not ready to add a second truth. You are ready to ask a better question of the first one.

Context

College systems, and before that travel and commerce products, grow clients the way ivy grows: first the site you own, then an app, then a script in operations, then a partner. Authentication is tokens or sessions. The server is Laravel. The consumers are not one framework and not one version.

I still like boring REST. I like it more when it is boring on purpose: stable URLs, stable error envelopes, pagination that does not reshuffle the universe, idempotency on the writes that hurt.

The other consumers you forget are your own jobs and your own admin. They encode the same assumptions. A "private" cleanup that only the SPA used will still break a command that has been posting the old shape on a schedule. Inventory callers, including the ones that share your repo.

The problem

The apparent problem is documentation. If we described the endpoints, people would use them correctly.

The real problem is time. Clients age in the wild. A phone that is not updated is still a production caller. A partner who integrated against your example payload will treat your example as law. Your undocumented assumption — missing field means zero, extra field is ignored, sort is by id — becomes their architecture.

When you "clean up" the API, you are asking all of those people to do a project. They will not all say yes this sprint.

The problem is also ownership. An internal endpoint has one team on both sides of the wire. An infrastructural endpoint has a population. You cannot hold a standup with a population. You can hold a deprecation window, a changelog, and a measurement of who is left. If you have none of those, you do not own the change. You own the surprise.

The tempting solution

Ship breaking changes behind "it's a new version" in the ticket, but not in the URL, and hope the only client is the one you deployed today.

Or version everything from day one, /v1 /v2 /v3, and carry three implementations of the same confusion.

Or refuse to document so nobody depends on you. They will depend on you anyway. They will depend on tcpdump.

Why that is not enough

Compatibility is not a header. It is a budget. Every field you expose is a promise. Every implicit default is a promise. Every 200 that used to mean "created" and now means "accepted" is a broken promise.

Idempotency is part of this. A client on a flaky network will retry. If create is not safe to repeat, you will meet them in support.

Pagination is part of this. Offset pages drift when writes happen underneath. Mobile lists feel haunted. You will blame the app. The contract was the ghost.

Auth is part of this. A token that never expires is easy. A token that expires without a refresh story is how you teach users to screenshot errors.

Validation is part of this in a way people underestimate. Tightening a rule — a field that used to accept a blank, a date that used to accept a time — is a breaking change even if the OpenAPI file still says the same name. Old clients were not sloppy. They were obeying last year's honesty. If you need the tighter rule, add a new field or a new route, or give them a window. Silent strictness is how you turn a cleanup into an outage for someone who did not deploy this week.

Options

Treat the API as private and force all clients to deploy with it. Advantage: freedom. Disadvantage: you do not control the store, the partner, or the forgotten script.

Additive change only, explicit deprecation, versions when the meaning of a resource actually changes. Advantage: you can keep shipping. Disadvantage: you carry dead fields. Dead fields are cheaper than dead apps.

A new API style — GraphQL, a Backend-for-Frontend per client. Advantage: clients fetch what they want. Disadvantage: you have moved complexity into a layer you must operate, and you still have compatibility inside the graph. I would add a BFF when one client is distorting the public contract, not as a fashion.

Trade-offs

You trade cleanliness for duration. The JSON will not photograph well. It will work in last year's phone.

You trade some delivery speed. Review now includes "who else calls this." That question is slower than "does the page work."

You should not trade away a way to see usage. If you cannot tell who is still on the old shape, deprecation is a mood. A counter on the old field is a cheaper conversation than a guess in a meeting.

Decision

I treat any endpoint a mobile client or an external party can reach as a product. Additive first. Rename as a new field plus a long goodbye. Errors in one envelope. Writes that create money or messages take an idempotency key.

I version in the URL when I cannot keep the meaning of a resource honest without lying to old callers. I do not version because a blog post said so.

I also treat undocumented behaviour as part of the contract once it has users. If clients depend on a sort order you never promised, you still own the tantrum when you change it. You can document it as undefined and still choose not to break it this quarter. That is not weakness. That is knowing the difference between a spec and a population.

Implementation / Thinking process

Write the contract as if a stranger will implement it on a train. Examples include the ugly extras: empty lists, partials, 409s. Strangers copy examples.

Keep pagination stable. If the list can move while someone scrolls, prefer a cursor. If you cannot, say that the list is a snapshot and live with the implication.

Validation errors should name fields the client sent, not your internal DTO. The client cannot fix $data['nested']['x'].

For auth, pair short-lived access with a refresh you can revoke. A mobile app that stores a forever token is a walking breach you scheduled.

When we must break, we do it with a date, a header or a path, and a log of who is left. The date is a product conversation. Engineering does not get to pick it alone if a store release is involved.

Idempotency keys need a home. If the client must send them, say so in the error when they do not, and store them long enough to cover retries, not long enough to become a second database of everything. If the server invents them, the client cannot help you.

Pagination should state what "the next page" means when a row is inserted above the fold. If you cannot say it, do not pretend the list is live and consistent. Mobile engineers will invent a meaning. That meaning will become your bug.

Think in populations, not in pull requests. A change that is compatible with the SPA you shipped today can still be incompatible with the binary from March, the job that posts a subset of fields, and the partner who coded against a screenshot. Compatibility review is "who is still out there," not "does our frontend compile."

When a field must die, give it a retirement: still accepted, still returned, documented as departing, then a date. The date is the product. Engineering can recommend one. Engineering cannot pick one alone if a store review or a partner contract sits in the way.

I also want one error envelope. Not a new JSON dialect per controller. Clients write one failure handler. If every endpoint invents a shape, they will swallow errors to look polished, and you will debug silence. The envelope is manners. Manners are the long-lived part of an API.

List what "done" means for a write. Created, accepted, already existed, conflicted. If those four share one 200 and a vague body, clients will guess, and their guesses will fork your product. Status codes are cheap. Ambiguous success is not.

Failure modes

A "small" nullability change. Old clients crash on missing keys. New clients assume missing means default. You have two sciences.

A success status that changed from 201 to 200. Someone's sync loop now thinks nothing was created and creates again.

Returning different shapes for the same resource based on who is asking, without saying so. That is not flexibility. That is a maze.

Documenting the happy path only. The undocumented branch becomes folklore, then a dependency.

Operational consequences

Once the API is infrastructure, your on-call includes other people's release trains. A deploy that is fine for the website can be an incident for the app. You will want contract tests that run the old fixtures, not only the new SPA.

You will also grow political weight. Saying no to a breaking cleanup is part of the job. People will call you conservative. They are describing the bill.

The other operational fact is archaeology. Six months after a field is "unused," a partner script will still send it. If you delete on a feeling, you will meet that script as a Sev. Keep a way to see the old shape arrive. A counter is enough. Feelings are not telemetry.

Lessons

A function you can change is code. A behaviour other software has encoded is infrastructure. The same route can be one on Monday and the other by Friday.

Compatibility is not kindness. It is how you do not turn your users into an unpaid QA department for your refactors.

What I would do differently today

I would have frozen error envelopes before I froze resources. Clients handle failure more often than they handle your new field.

I would have added a consumer list — web, iOS, Android, jobs, partners — next to any route change in review. If the list is "I think just us," it is incomplete.

I would have treated pagination and error envelopes as the first frozen surfaces, not the resource names. Clients recover from a new field. They do not recover from a list that reshuffles or a 422 that changed its clothes. Freeze the manners early. Rename the nouns when you must, additively.

Closing thought

The day another programme depends on your JSON, you are in the compatibility business. You can still change. You just have to pay for the change in time and dual-running, not in surprise crashes on a phone you cannot reach.

Context

The system in my head here is a monolith that began in an older Laravel and grew into the operating layer of a real organisation. Student-facing apps, staff tools, finance edges, a mobile client that could not be forced to a flag day. Features did not stop arriving because the framework documentation had moved on.

The team was small and already busy. That is the usual setup, and it is the one rewrite plans like to forget. A rewrite is often estimated as if a second team existed. In practice the same people would have been asked to keep the old product alive, answer production, and invent the new one with better habits.

Users were not internal-only. If we broke login or a payment return URL, we would hear it from people who do not care what a service provider is. That raises the quality bar on temporary dual running. Temporary, in these systems, has a habit of becoming the architecture.

The real problem

The apparent problem was technology age. Unsupported PHP. Packages that had died. Framework conventions that made new work feel like translation. Those are real costs. They are also the costs that are easiest to put in a slide.

The deeper problem was that the old system was the only complete description of the business. Not a good description. A complete one, including the exceptions. A rewrite that starts from interviews will get the official process. It will miss the unofficial one that operations invented because the official one never fitted. Then you cut over and discover that the new system is correct and unusable.

There is another deeper problem: rewrite versus refactor is the wrong binary. The honest options are about blast radius and time. Can you move the runtime under the product without inventing the product again? Can you extract a boundary while the old one still serves? Can you keep shipping the work the organisation is already committed to, or are you asking them to live in a museum for a year?

If you cannot answer those, you are not choosing architecture. You are choosing a story about a future team that does not have your calendar.

Constraints

The product had to keep taking money and keeping sessions. That constraint is not romantic. It eliminates big-bang cutovers that depend on a long freeze, and it eliminates rewrites that only work if feature work stops.

Mobile clients and email links extend the past. You can deploy a server every day. You cannot force every phone to upgrade on Tuesday. Compatibility is not a slogan here. It is a window measured in months.

Data quality was uneven. A new schema that refuses the old dirt will fail at import, then fail again when someone finds a row that was legally meaningful and technically illegal.

Budget was attention more than money. A small team has a finite number of careful hours. A rewrite consumes those hours in a place users cannot see, while the visible product still demands them.

Third-party callbacks — payments, sign-in, inbound mail — had URLs and assumptions. A new application that wants a clean slate still inherits those promises unless you enjoy coordinating every vendor at once.

Options considered

Rewrite in parallel, cut over later. Advantage: psychological freshness, a chance to drop rituals nobody can defend. Disadvantage: you fund two products and a translation layer. Risk: the new product ships the brochure and misses the exceptions; the cutover becomes a second project with its own politics. Maintenance doubles. The business sees a year of "nearly" while the old system still holds the truth.

Freeze features and renovate in place. Advantage: focus. Disadvantage: the organisation does not freeze. Someone will need a change that is not renovation. Risk: the freeze is a fiction, or it holds and you lose trust. Complexity is social more than technical. I have rarely seen this survive contact with a term date.

Strangle one seam at a time while the monolith stays the system of record. Advantage: each step can be reversed; users stay on one product. Disadvantage: you live with seams, flags, and a codebase that looks indecisive in the middle. Risk: the seams become permanent suburbs. Maintenance is uglier in the short term and cheaper than a failed rewrite if you actually finish seams.

Version-walk the existing app, keep the domain, replace only what is dead. Advantage: you keep the behaviour you already paid to discover. Disadvantage: you will carry some shameful code into a modern runtime. Risk: you confuse a framework upgrade with a design improvement and declare victory too early. This is the least glamorous option and the one that usually respects the constraints.

I should say what I rejected quickly: a microservices conversion as the migration strategy. Taking an unclear monolith and spraying it across a network gives you an unclear distributed system. That is not courage. It is a way to make the same coupling more expensive.

Decision

We version-walked the monolith and reserved extraction for places where a boundary was already true: a queue-heavy email path, a payment callback, a read-heavy report that was harming writes. We did not start a second product.

I chose that because the organisation needed the old behaviour more than it needed a new folder structure. The framework work was justified on security, hiring, and the cost of standing still. The rewrite was not justified, because we could not staff the truth twice.

The "why" was also personal in the professional sense. I had seen enough multi-tenant and back-office systems to know that the second system is where optimism goes to meet the exception list. I would rather be accused of conservatism than of disappearing for six months and coming back with a demo that cannot invoice.

We still said no to some features that would have deepened a rotten seam. That is not a freeze. It is triage. If a request would only make the later version-walk harder, we offered a smaller change or a later date. That is a tech lead decision wearing an architecture costume.

Implementation / Process

The path was a sequence of boring contracts. First, make the app build and test on a runtime we could still patch. Then move one major framework version at a time, with the official upgrade notes as a checklist rather than as bedtime reading. Then replace packages that could not come with us, preferably with something dull.

We kept shipping inside that sequence. The rule was that a feature had to land on the current production line, not on an imagined future line. If a change was easier after the next version step, we sometimes waited a short, named period. We did not open a forever branch.

Where we extracted, we extracted at an existing seam. Email already wanted a queue. Payments already wanted idempotent workers. Those could become clearer modules without pretending they were new companies. The monolith remained the place sessions and permissions made sense.

Deployment stayed reversible. Expand/contract for schema. Feature flags only when we could say who would turn them off. A migration that cannot roll forward in small steps will eventually be attempted as a weekend, and weekends are when you discover the exception list.

Testing followed risk, not ideology. Auth, money, tenancy, the jobs that send the organisation's voice to the outside world. We did not wait for a perfect suite. We also did not walk versions on hope.

Problems and failures

We under-priced compatibility. A mobile client or a stored link will keep calling a shape you have already grown tired of. I still catch myself designing the next server as if clients were polite.

Package replacement took longer than version bumps. The framework is documented. The abandoned package that encoded a local tax or a forgotten SSO path is not. That is where rewrites look attractive: you imagine you will not have to understand the package if you throw it away. You will have to understand the behaviour, which is worse.

We had a mid-project moment of rewrite temptation after a particularly ugly module. The correct response was to extract that module's tests and shrink it, not to declare the programme dead. I did not always give that response on the first day. Fatigue makes people philosophical.

Communication failed when we talked about versions instead of risk. "We are going to Laravel N" is not a stakeholder sentence. "We are removing a class of security and hiring risk while keeping checkout intact" is. I had to learn that translation again, even though I already knew it.

There were production surprises that were not the framework's fault. They were ours: a config cache, a queue worker that still had the old code in memory, a scheduled command nobody had run in staging. Migration problems are often operations problems wearing composer clothing.

Trade-offs

We accepted a codebase that would not photograph well for a while. Mixed conventions, shims, a few adapters that existed to protect the rest of the app from a dead library. That is uglier than a new repo. It is also a product that still knew how to do Tuesday.

We spent calendar time that could have been features. That cost is real. I do not hide it. The opposing cost is a security story or a hiring story that arrives on a deadline you do not choose.

We did not get a pure domain model. We got a runnable modern runtime and a few honest seams. Anyone who needs purity should not take responsibility for a live back-office. Purity is a luxury of the first version, or of a product that has not yet met its users.

Result

The organisation did not experience a cutover day, which I take as the point. People kept logging in. Money kept attaching to invoices. We were able to hire and onboard against current docs instead of folklore from a dead major version.

The codebase was still itself. Anyone expecting a different personality would have been disappointed. What changed was the cost of the next change and the shame level of the platform conversation. Those are dull victories. They are the kind that compound.

I will not invent a downtime number. The honest result is negative space: the migration did not become the only story of that year.

What I would do differently today

I would have written the "what this system actually does" inventory before the first version bump, not after the third ugly package. Behaviour first, runtime second. I knew that and still let composer set the agenda because composer is concrete.

I would have been earlier and kinder about the no. Some features we accepted made the walk harder and were not material to the term. A clearer intake rule would have saved more engineering time than a clever shim.

I would also have rehearsed worker restarts and schedule overlap as first-class migration steps. We treated them as deploy hygiene. They are where users feel a migration that "went fine" on the web nodes.

Broader lesson

A rewrite is a claim that you understand the business well enough to discard its only working description. Sometimes that claim is true. In a live operational product, it is usually vanity.

The design problem is the path: how behaviour survives contact with a new runtime, a new package set, a new team. If your plan has a destination and no path the calendar can live with, you do not have a strategy. You have a wish that will be re-estimated in six months.

Closing thought

Move the ground under the product if you have to. Do not ask the product to levitate while you build a better one in the next room. The business will not pause to applaud the architecture, and the exceptions will not volunteer to be documented.

Context

The products were Laravel and MySQL, sometimes Redis for cache and queues, one deployable. Clients expected their data to be theirs. They did not all expect a private database. Early on, neither did we. Shared schema is how a small team ships a marketplace or a school system without becoming a hosting company.

The pressure arrives in pieces. A report that should not see another school's students. A job that forgot the tenant and emailed the wrong list. A customer who wants a restore that is not everyone else's restore. A noisy tenant whose catalogue search makes the shared primary unhappy.

None of those is theoretical. They are the ordinary weather of this shape.

I also include reporting as a first-class pressure. A platform that needs a cross-tenant view — usage, billing, support — will fight hard isolation. A customer who needs a legal export of only themselves will fight hard sharing. You cannot satisfy both with a slogan. You have to say which one you are building this year.

The problem

The apparent problem is how to store many customers in one app.

The real problem is blast radius. Who can read. Who can write. What happens when a query is wrong. What happens when a migration is slow. What happens when you need to give one customer their data and not a neighbour's. Tenancy is those questions wearing a schema.

A useful way to say it: tenancy is not a feature you add to an app that already works. It is a property of every place work enters and every place data leaves. HTTP is one entrance. The queue is another. The scheduler is a third. A report export is an exit. A support impersonation tool is both. If you only designed the first, you designed a lobby and left the loading dock unlocked.

The tempting solution

Add tenant_id, a global scope, middleware that sets the current tenant from the host or the token. Feel finished. Most tutorials end here because the demo only has one happy request.

If you are ambitious, you sprinkle tenant_id on every table "just in case," including tables that are actually global, and then you spend a year explaining why a country list is per client.

Why that is not enough

Request middleware does not run in the console, in a queue, in a webhook, in a broadcast callback, unless you make it. The leak I worry about is not the clever attacker first. It is the forgotten job.

Global scopes are helpful and treacherous. They hide the predicate. A new query that uses a raw join or a different connection will not hear the sermon. Tests that do not include a second tenant will not either.

Shared hardware means noisy neighbours. One tenant's import is everyone's Saturday. Isolation of data is not isolation of load.

Backups and legal stories diverge. "We take a dump of production" is not an answer to "give us our data as of Thursday." If you promised surgical restore, you bought a different architecture than you drew.

There is a quieter leak: metadata. Logs, exception trackers, search, a support tool that stores the last request. If those systems do not know tenancy, you will paste a payload into a ticket and have shown one customer another customer's life. Isolation that stops at Eloquent is costume isolation.

Options

Shared schema, tenant_id, hard scopes, adversarial tests. Advantage: one migration, one report path, operable by a small team. Disadvantage: a class of bugs that are catastrophic rather than local. Risk: you rely on culture. Culture is not a control.

Shared database, separate schemas per tenant. Advantage: slightly clearer isolation, still one server. Disadvantage: migrations become a fleet problem. You will write a runner. The runner will be a product.

Database per tenant. Advantage: restore, noisy neighbour, and some compliance conversations get easier. Disadvantage: connection management, schema drift, cost, and a platform you must actually run. I have seen teams jump here for prestige and drown in plumbing.

Separate application per tenant. Advantage: true isolation. Disadvantage: you no longer have a multi-tenant product. You have a services company.

I start shared, with hostility toward implicit context, and I move a tenant to stronger isolation only when a concrete force — restore, load, contract — pays for the platform.

There is a fifth option people reach for under panic: a query parameter that "the admin can pass to see another tenant." That is impersonation. Impersonation is a product with an audit. It is not a debug flag you leave in the report controller.

Trade-offs

Shared is cheaper to evolve and more expensive to get wrong. Isolated is more expensive to evolve and cheaper to explain after an incident.

Reporting wants shared. A single query across tenants is how a vendor operates. Isolation wants you to build an export warehouse. That warehouse is often the real second system.

You should not trade away a fail-closed default. Unknown tenant means no data, not "first tenant we found." A missing context in a job is a crash I will take. A guessed context is a leak I will not.

Decision

For the products I have been responsible for, shared schema was the honest start. The decision that mattered was not the column. It was that tenant context had to be explicit in every non-HTTP entry point, and that a query without a tenant was a bug we could write a test for.

I would not pick database-per-tenant because it sounds mature. I would pick it when a customer or a load profile makes shared restore or shared CPU indefensible.

Hybrid is allowed. One noisy or regulated tenant on their own database, the rest shared, is an operational shape. It is also a complexity tax. If you take it, write the rules for migrations and reporting or you will grow a special case that only one person can deploy.

Implementation / Thinking process

Resolve the tenant at the edge. Bind it once. Pass it into jobs as data, not as a leftover singleton you hope is still there. I have lost time to workers that remembered the previous tenant. That is a horror story you can prevent with boring parameters.

Use scopes, yes, and also use tests that create two tenants and try to read across. If your suite never has a second tenant, you do not have tenancy tests. You have happy-path tests with extra paint.

Cache keys include the tenant. Queue names sometimes should. Rate limits almost always should. Otherwise you have isolation in MySQL and a commune in Redis.

Think about the admin path. A platform operator who can see all tenants is a different role than a tenant admin. If you reuse the same gates, you will one day show the wrong school in a typeahead.

Search, mail, and files are where tenancy goes to die. A global index, a shared bucket with guessable keys, a mail job that loads recipients without a tenant predicate — the relational scope will not save you. If the artefact leaves MySQL, it needs the tenant in its name or in its query, or it is public with extra steps.

Tests should try to be hostile. Create tenant A, act as tenant B, assert empty. Do it for HTTP, for a job, and for a report. If any of those is missing, that is the next leak.

Failure modes

The unique index that forgot the tenant. Two clients cannot create "ADMIN" or "DEFAULT." You "fix" it by suffixing, which is a smell that the identity was never tenant-local.

A search index or a Scout driver that is global. The database was scoped. The search was not.

A report "for us" that runs without context because it runs on a schedule. Schedulers are tenants too, or they are superusers. Pick one and write it down.

A migration that locks a hot shared table. Isolation of rows does not isolate DDL.

The helpful typeahead that searches people globally because the developer was building support tools. Support tools are the most dangerous clients you have. They are staffed by people you trust, pointed at every tenant, and rarely covered by the same tests as the customer path.

Operational consequences

Support will ask you to "just look at tenant X." Your tools will grow a tenant switcher. That switcher is production infrastructure. If it is a raw database change in a query window, you will eventually run the query in the wrong place.

As you grow, someone will request an exception: this customer needs their own database. If you have no story for a hybrid fleet, you will either refuse a real need or invent a one-off that becomes the new architecture by accident.

Billing and support will also ask for cross-tenant search. That search is a privileged product. If you build it as "the same query without the scope," you have implemented the leak on purpose. Build it as a separate capability with an audit, or do not build it.

Lessons

Tenancy is an isolation budget. Spend it on the failures you cannot tolerate. Do not spend it on a diagram.

A column is a field. A boundary is a property of every way work enters the system. Only one of those is multi-tenancy.

What I would do differently today

I would have written the two-tenant test before the first global scope. Scopes make you feel safe. Tests make you safer.

I would have treated Redis keys as tenanted from day one. I used to fix that after a surprise. The surprise is always the same: a cache is a second database with worse manners.

I would have written the restore story before the sales story. "We can give you your data" is easy to say in a shared dump. It is a different product if it must be theirs and only theirs, as of a day, without a week of engineering. If you cannot keep that sentence, do not offer it. Offering it is how shared-schema teams inherit an isolation architecture they never budgeted.

Closing thought

You can run many clients on one schema for a long time if you are willing to make leakage hard and to say no to restore stories you cannot keep. What you cannot do is remember the tenant. Memory is not an isolation primitive.

Context

The applications were ordinary Laravel products on MySQL, with Redis already in the building for cache and queues. They were not hyperscale. They were busy enough that a bad query was felt by staff who used the same screens all day, and by students or agents who hit a small number of hot paths. Volume was uneven. Enrolment windows, payment deadlines, reporting days. The system was fine on a quiet Tuesday and rude on a day that mattered.

The team reflex toward infrastructure was understandable. Vertical scale is a ticket. It has a vendor page. It does not require arguing about an Eloquent relation in a ten-year-old module. Application work requires someone to own a slow path, and slow paths live in the least fashionable code.

We also had the usual monitoring: instance CPU, memory, disk IO, a few application timers. What we did not always have was a habit of asking which query, for which screen, with which parameters, caused the room to feel slow. Without that, every graph becomes a Rorschach test. You see what you already wanted to buy.

The real problem

The apparent problem was capacity. The box was working hard. Connections stacked. A replica sounded mature. Cache sounded responsible.

The deeper problem was amplification. One user action became dozens or hundreds of queries. A list endpoint loaded children, then grandchildren, then a helper that ran a count per row. A dashboard recomputed a permission-heavy report that was identical for every staff user in the same role for minutes at a time. A search that allowed "just one more filter" defeated the only index that had been doing real work.

There is a second deeper problem: we often cache as a way of not admitting the read model is wrong. Redis will happily store a blob that took two seconds to build. Then invalidation becomes a folklore project, and you have a fast lie or a slow truth, depending on the day. I like Redis. I do not like it as a witness protection programme for a query we are afraid to look at.

Infrastructure can still be the answer. If the working set no longer fits, if you are CPU-bound on honest work, if a replica would isolate reporting from checkout, buy the thing. The failure is buying it so you can postpone naming the query.

Constraints

We could not take the hot screens down for a rewrite. Staff would simply export to CSV and leave, which is a worse architecture than a slow page.

We could not pretend we had infinite time to perfect every list. Some reports were going to remain heavy. The decision was which heaviness was honest and which was accidental.

Schema change was possible but not free. An index is a write tax. A new table for a read model is a consistency tax. Both are cheaper than a class of instance you will be shy to downsize later, but they still need owners.

Cache was already in play, which meant some of the worst paths were hidden except when they missed. A cache hit is not proof of a good design. It is proof that yesterday's answer was reusable.

And we had to keep mobile and web clients working. If the fix was "the API now returns less," the clients had to be able to live with less, or we had to add a dedicated shape instead of starving the old one.

Options considered

Scale the primary. Advantage: sometimes it works immediately, which is a hell of a drug. Disadvantage: you have not reduced work, only the time each unit of work takes, until growth eats the gift. Risk: the next busy period returns with a larger invoice and the same queries. Maintenance of the application stays the same. The business pays rent for a shape nobody improved.

Add a replica and send reports there. Advantage: this can be the right isolation when reporting and checkout truly fight. Disadvantage: replication lag becomes a product bug the first time someone pays and then opens a report. Risk: you copy a bad query to a second machine and call it architecture. Complexity in routing and failover is real. I would do it for isolation, not for shame.

Cache the world. Advantage: the next request is cheap if you get the key right. Disadvantage: you now own invalidation, stampedes, and the support ticket that says the number is wrong. Risk: you cache a permission-sensitive payload too broadly, or you cache too narrowly and get no hit rate. Maintenance shifts into a second correctness problem. Business impact is ugly when a cached dashboard disagrees with a live invoice.

Fix the access path. Narrow selects, eager load what you actually need, stop N+1, give the list a cursor or a cap, push a truly expensive report into a job and a stored result. Advantage: the database does less, which is the only scale factor you own forever. Disadvantage: it is slower to show in a procurement meeting. Risk: you micro-optimise the wrong screen. Complexity is local if you stay honest. Maintenance usually improves because the code starts stating its data needs.

Build a small read model for one hot view. Advantage: you can index for the page you have, not the relations you inherited. Disadvantage: you write twice, or you project asynchronously and accept delay. Risk: the projection lags and people trust it anyway. This is the option I reach for when the screen is important and the relational truth is the wrong shape for it.

Decision

We did not start with a bigger instance. We started with a rule: no infrastructure ticket without a query story. Which request, which SQL, how often, what it did in EXPLAIN, whether the application issued it in a loop. If we could not say that, we were not ready to spend money.

Where the story was an N+1 or a missing composite index that matched a real filter, we fixed that first. Where the story was a dashboard that rebuilt a universe, we stopped rebuilding it on every request. Where the story was a report that belonged in the background, we took it off the request/response path instead of giving MySQL more CPU to fail more expensively.

I chose that order because I had already watched hardware hide bugs. A faster disk makes a tablescan less embarrassing. It does not make it a strategy. I also chose it because a small team feels every recurring cost. An instance you do not need becomes a quiet tax on every other decision.

We left the door open to scale. That was important. The decision was sequence, not religion.

Implementation / Process

We treated the slow path like an incident with a lower pulse. Capture the endpoint. Turn on enough query logging in a controlled way to see the shape, not to drown. Count queries on the page in development until the number stopped being a joke. Read the plan. Look at cardinality instead of guessing that an index on status would save us when status had three values and the filter was something else.

For lists, we asked a rude question: does anyone need the tenth thousandth row in this UI? If not, the UI had been lying about being a list. It was an export wearing a table tag. Exports can be jobs.

For dashboards, we asked whether the number had to be live to the second. Most operational numbers do not. They have to be trustworthy and fresh enough. A computed snapshot, rebuilt on a schedule or on the writes that matter, is often the whole architecture.

Redis entered as a specific answer, not a mood. Cache a snapshot that is safe to reuse, with a key that includes the tenant and the meaning of the number, and an invalidation path you can describe in one sentence. If you cannot describe the invalidation, you do not have a cache. You have a rumour with a TTL.

We did not need a new service. We needed the monolith to stop being shy about its reads.

Problems and failures

The first failure was local optimisation. We made one admin list respectable and left a related export that still walked the same graph. Users switched to the export and the database stayed busy. Performance work that ignores the escape hatches users already have is decoration.

The second was an index we added because it looked official. It helped a report we run rarely and slowed a write path we run constantly. Unused or wrongly ordered indexes are not free. They are a permanent comment in InnoDB.

Cache created a trust incident. A count was right enough until a permission changed and the key did not know. The number was fast and wrong. Fast and wrong is how you train people to ignore the dashboard, at which point you have paid for a UI that nobody believes.

We also wasted time arguing from production feelings. "It felt worse after the deploy" is a clue, not a measurement. Until we could replay the endpoint, we were just swapping anecdotes. That is an engineering process failure, not a MySQL one.

Communication with non-engineers was clumsy. "We are not upgrading the database" sounds like refusal. "The page is asking the database the same question two hundred times" is a sentence people can dislike for honest reasons. I learned to lead with the second.

Trade-offs

We chose application complexity over vendor simplicity in a few places. A snapshot table is more to maintain than a larger RDS class. I still prefer the snapshot if the team can own it, because the cost stays visible in the codebase instead of in a bill nobody revisits.

We chose slightly stale operational numbers over live-but-random latency. That is a product decision dressed as a technical one. Some rooms need live. Most rooms need calm.

We did not choose purity. Some paths stayed cached and a bit ugly. Some indexes stayed because dropping them felt riskier than proving they were useless. Sequence does not require zeal.

Result

The useful result was that a few screens stopped being folklore. People could open them on a busy day without a ritual. The database still worked hard when the business was actually doing a lot. That kind of hard looks different in the graphs: it is fat work, not frantic work.

We still scaled other things later when the case was honest. I do not consider that a contradiction. The point of refusing the first bigger-database conversation was to make the later ones boring and specific.

I will not quote a latency number I did not keep. I will say the team started bringing EXPLAIN plans to discussions that used to start with instance types. That was the cultural result I actually wanted.

What I would do differently today

I would have instrumented the hot endpoints before the first panic, not during it. Adding timing and query counts when people are already angry makes the work feel like blame. Doing it in peacetime makes it a dashboard.

I would have been harsher about unbounded admin lists. We kept a few "load everything" views because a stakeholder liked to scroll. That is not a requirement. It is a habit, and it is expensive.

I would also have separated "reporting MySQL" from "we need a replica" earlier as a language habit. Those are not the same project. One can be a job writing to a table. The other is topology. Mixing the words makes people buy topology.

Broader lesson

Infrastructure is a way of paying for work you have already agreed to do. Architecture is a way of refusing work that should not exist. When a system feels slow, ask which of those you are looking at.

If you cannot name the query, you are not ready to name the instance. If you can name the query and it is honest, then buy the machine and do not perform a morality play about it.

Closing thought

A bigger database is sometimes the right purchase. It is almost never the right first sentence. Make the application tell you what it is asking. Then decide whether you want to ask less, ask less often, or pay more to keep asking the same thing.

Context

Most of the products I work on have more than one mouth. A staff SPA or Blade-heavy admin. A student-facing site. A Flutter app. Jobs that write the same records. Sometimes a partner. The backend is Laravel. The frontend stack has been Vue, React, a bit of both, depending on the surface.

That mix is ordinary now. It makes the old habit — "we will validate in the form, the API is for us" — more expensive every year. "Us" keeps growing.

I include Blade admin pages in this. They feel like the server, so people put rules in the template or in a livewire-ish click and skip the policy. A server-rendered hide is still a hide. The next JSON client will not read the template.

The problem

The apparent problem is duplication. We already check the field in the UI. Why again on the server. It feels slow and pedantic.

The real problem is authority. If the rule exists only in the screen, it exists only for people who use that screen, in that version, in that browser, with that feature flag. Everyone else is on a different product that happens to share a database.

There is a second problem: the UI is a terrible archive. Six months later nobody remembers why Sunday was blocked. The picker is just how the form feels. The server, if it never knew, cannot tell you.

There is a third: time. A rule that lived in a release of the SPA is a rule that dies when the bundle is replaced. Jobs and old mobile builds do not get the new bundle. If the only copy of the law was in JavaScript, you have a legal system with a cache-busting problem.

The tempting solution

Put the cleverness in the client because that is where the friction is felt. Disable the button. Prefill the only legal option. Trust that "nobody uses the API directly."

Or the opposite extreme: dump every rule into the JSON and make the UI a stupid renderer. That sounds pure. It also produces chatty endpoints and a frontend that cannot sketch a decent empty state without a round trip.

Why that is not enough

Clients are not a security boundary. Hiding a button is not authorisation. I have seen "we do not show delete" stand in for "delete is not allowed." It is allowed. You just made it a privilege for people who can open a console.

Validation that only lives in the UI also fails closed for good users and open for everyone else. The honest student hits a red border. The import job writes garbage.

And frontend assumptions compound. A status colour becomes a meaning. A default tab becomes a workflow. A local filter becomes "the list of people who need attention." Support starts speaking in UI nouns. Then you change the UI and the business thinks you changed the policy.

I have watched a colour — amber, green, a badge — become the only way a team talks about risk. The API had a status. The colour was a designer's mapping. When the mapping changed, people thought the business rule changed. That is not a design nit. That is an unwritten policy with a hex code.

The same happens with defaults. Prefill "this campus" because it is usually right, and you will create a year of records in the wrong place for the person who did not notice. A default is a decision. If the server does not know it happened, you cannot audit it.

Options

UI as the only rule-keeper. Advantage: fast, tactile. Disadvantage: every new client reimplements the company. Risk: the admin bypasses it on day two.

Server as the only intelligence, UI as a dumb shell. Advantage: one place to test. Disadvantage: poor interaction design, extra latency, and a temptation to over-fetch.

Shared rule, different jobs. The server decides what is legal. The UI decides how to teach, how to sequence, how to recover from a no. Duplicate the cheap checks for courtesy — required fields, format — and never the expensive ones: eligibility, money, tenancy, who may approve. Advantage: users get a kind screen and a honest API. Disadvantage: you must keep the courtesy checks from drifting into a second policy.

Generate the UI from a schema. Advantage: less drift. Disadvantage: the worst forms I have used were honest and unusable. Schema-driven UI is a tool, not a product strategy.

A "may I" endpoint plus a write that re-checks. Advantage: the screen can be kind without being the law. Disadvantage: two calls. I pay that on eligibility, money, and approval. I do not pay it on whether the email field looks like an email.

Trade-offs

You accept some duplicated validation. A required email in both places is cheap. A pricing rule in both places is how you get two prices.

You accept that some UI behaviour is not a rule. Animation, density, which field is first. Those belong to the client. If you pull them into the API you will version button padding.

You should not accept a 200 from the API for a state the product considers impossible. That is how jobs and future clients become attackers.

Decision

The API decides. The UI explains. I want the server to reject the illegal write with a contract a client can display. I want the screen to make the legal path obvious so we do not use error messages as the design.

When I review a frontend change that "just hides" a dangerous action, I ask where the same hide exists for the job and the mobile client. If the answer is nowhere, we have not implemented a rule. We have implemented a vibe.

I am willing to duplicate cheap checks. Required, format, "pick one of these four." I am not willing to duplicate eligibility, price, or "may approve." Those belong in one function the UI may call and the write must call. If the UI cannot afford the round trip, the action was not ready to be a single tap.

Implementation / Thinking process

I keep write validation at the application boundary — form requests, command objects, domain services — not only in Vue components. The UI may call a lightweight "may I" endpoint when the flow is expensive to get wrong, but the write still checks.

Error shapes matter here. If every failure is a 422 with a human sentence and a stable code, the web and the phone can behave like they work at the same company. If every controller invents a JSON dialect, the UI will start swallowing errors to look polished.

I let the frontend own interaction state: wizards, dirty forms, optimistic lists that reconcile on failure. I do not let it own eligibility. Optimistic UI that assumes the server will agree is a good animation and a bad accountant.

Shared TypeScript types or OpenAPI help. They do not replace tests on the write path. Types describe the conversation. Tests describe the law.

I also want the frontend to receive reasons it can show. A boolean "not allowed" trains designers to invent copy. A code plus a sentence lets the same rule serve web and mobile without a second policy document in Figma.

Watch local state that outlives the server. A wizard that keeps "approved" in memory and then posts a different resource is how you get a UI that thinks it finished. On submit, the server is the story. The wizard is a draft.

Failure modes

A "temporary" admin screen that talks to Eloquent directly. It will be the real product on the last day of the month.

A mobile build that caches an old allow/deny. The user is not hostile. They are on a train with last week's binary.

Frontend tests that assert the button is disabled, with no server test that the action is rejected. You have tested the wallpaper.

A designer-led flow that adds a step the API cannot see — "I confirm I checked the passport" — stored only in component state. The organisation thinks it has a control. It has a click.

Operational consequences

Support will file tickets against the screen. Engineering will look at the API and say it is fine. Both can be right. The organisation is running two products. The meeting will be about tone. The fix is to put the rule in the place both clients already obey.

When you finally move the rule to the server, someone will complain that a script they loved is broken. That script was the second client. Thank them. They found your missing test.

You will also find that some "frontend rules" were never rules. They were taste. Those can stay in the client. The work is to notice which is which before the taste calcifies into a meeting about "why did you allow that."

Lessons

The interface is allowed to be stricter than the API in the sense of being kinder. It is not allowed to be the only place the company says no.

If a second client would be able to do the thing, you do not have a business rule. You have a layout.

What I would do differently today

I would have added a second, ugly client earlier — a command or an admin action — in review, just to see whether the rule survived. One screen is a story. Two writers are a contract.

I would also have stopped treating "we will do it in the frontend first" as a phase. It is a fork. Forks do not merge themselves.

I would have treated admin "shortcuts" as clients. The ugly screen that posts to Eloquent is the most honest consumer you have. If it can violate the rule, the rule is optional. Make the shortcut go through the same write path, or admit you have two products and stop being surprised when they disagree.

Closing thought

A good UI teaches the path. A good API refuses the wrong one. If you only have the first, you have a product for the people who click where you expected. Everyone else is in a different business that shares your tables.

Context

The systems I work on lean on jobs the way operational products always do: webhooks, documents, sync, notifications, imports. Laravel queues, Redis, workers. The user-facing app stays snappy. The real work happens in a process that can die, restart, and run again without asking.

That is a good shape. It is also a shape that makes it easy to confuse "we have a queue" with "we thought about failure." The first is a dependency. The second is design.

I am not talking about email volume here. Mail has its own operational essay. I am talking about the generic habit: import this, sync that, notify them, generate the PDF, call the bank. The same worker personality shows up in all of them — hopeful, under-specified, and slightly proud of the retry count.

The problem

The apparent problem is time. The request is too slow if we do the work now. So we dispatch.

The real problem is that the work now has a life of its own. It can run after the user has left. It can run after the record has changed. It can run after a colleague already did the thing by hand. It can run twice because the worker died after the side effect and before the acknowledgement.

If you have not said what those situations mean, the queue will pick a meaning for you. Its meaning is usually "try again," which is only correct for a subset of failures.

There is a quieter version of the same problem: the job that succeeds against a record that is no longer eligible. The worker did not fail. The domain did. A retry policy that only looks at exceptions will keep performing favours for ghosts — a cancelled booking, a withdrawn application, a person who is no longer in the cohort. Success is not the same as still being the right work.

The tempting solution

Set tries to 3, add backoff, maybe a failed_jobs table. Feel responsible. This is what the framework makes easy, and ease is how defaults become doctrine.

If the job talks to a provider, wrap the call in a generic retry helper. Timeouts become retries. 500s become retries. Unique constraint errors become retries. Everything looks like weather.

Why that is not enough

Failures are not one species. A network timeout might be safe to repeat if the other side is idempotent, and disastrous if it already created a customer. A validation error will not become true because you waited a minute. A lock timeout might want a retry. A "this person is not eligible" should not.

Ordering is another lie we tell ourselves. Many queues do not preserve the story you had in your head. A later job can run first. If your handler assumes it is the only writer, you will invent last-write-wins bugs that only appear when the queue is busy.

User expectation is the third gap. The UI said "we will email you" or "we are processing." That sentence has a half-life. A job that succeeds tomorrow morning may be correct in the database and wrong as a product.

Observability is the fourth. A growing failed_jobs table is not a strategy. It is a graveyard with a count.

The fifth is fan-out. One user action becomes five jobs. Four succeed. One dies. The UI is green because the request was. The organisation is not, because the fifth job was the one that talked to finance. Retries on the four will not help. You need a composition: either the work is one intent with steps, or the UI must not lie until the set is done. Most teams do neither and call it eventual consistency, which is a fine phrase for a bad afternoon.

Options

Do the work in the request and accept the latency. Advantage: simpler failure, the user is still there. Disadvantage: you couple their patience to a vendor. Sometimes still right for tiny, safe writes.

Fire and forget with retries. Advantage: fast to write. Disadvantage: you have outsourced meaning to a counter.

Intent first, then a worker that can conclude: done, retry, dead, or needs a human. Advantage: the job describes work against current truth, not a snapshot that aged in Redis. Disadvantage: you write more state. This is the option I want for anything that creates money, messages, or records another system will treat as real.

A workflow engine. Advantage: visibility. Disadvantage: a second product. I would rather have a boring intent table than a platform we cannot staff.

Synchronous for the first side effect, async for the rest. Advantage: the user sees the dangerous part finish. Disadvantage: you have split one intent across two clocks. Only do this if the first part is small and the second part has its own visible state. Otherwise you have invented a half-queue.

Trade-offs

You give up the fantasy that async is free. You buy a state model and some dashboards. That is cheaper than duplicate side effects, but it does not look like velocity in a sprint review.

You also give up some throughput. Checking "have we already done this?" costs time. Not checking costs reputation.

What you should not trade is a way for a human to finish or cancel the work. Some jobs will die for reasons the code cannot honourably guess.

Decision

I want every job that can hurt to answer three questions before it is allowed to retry: is this failure transient, is the work still the work, and is the next attempt safe if the last one partly succeeded.

If I cannot answer those, I do not want a retry. I want a dead letter and a person, or I want the work redesigned until the answers exist.

The payload should be an identifier, not a novel. Reload the record. The world may have changed. A job that carries a stale "please mark paid" is how you fight the user.

I also want a visible "we are still working" that can expire. If the UI said processing, there is a time after which the honest sentence is failed, not still in the queue. Leaving people in processing forever is how you get duplicate taps and a second job you then have to make idempotent under pressure.

Implementation / Thinking process

Classify operations. Reads and webhooks you control can often retry. Creates against someone else's API need an idempotency key you persist, or a lookup before create. Updates need a version or a "still in the state I expected" check.

Give up on infinite hope. A bounded retry with jitter is a courtesy to weather. After that, the job is a case. Cases need owners, not more backoff.

Log the conclusion, not only the exception. We retried because the provider timed out. We stopped because the person is no longer eligible. Those sentences are how the next engineer debugs at speed.

Do not share one queue and one retry policy across interactive and bulk work. I have made that mistake in mail. It is the same mistake in any mixed workload. Urgency isolation is part of failure design.

Timeouts are part of the decision. A five-second HTTP timeout plus three retries is a twenty-second lie if the provider already started the work. Align timeout, idempotency, and user-facing time or you will retry yourself into duplicates while the user is still watching a spinner on another device.

Dead letters need a human path that is not "look at the table when you remember." A count, an age, an owner. Otherwise failed_jobs is a museum.

I also want the job to be allowed to do nothing. A no-op that records "already complete" is a success. A throw because the row is gone is a different success if the domain says the work is obsolete. Treating every empty result as a retry is how you invent storms. The handler should have a vocabulary: done, obsolete, retry, dead. Four words. Most queues only taught us two: throw or return.

Failure modes

Poison messages. One bad payload blocks a worker or retries forever and starves healthier work. Quarantine is a feature.

At-least-once delivery plus a non-idempotent create. You now have two invoices or two messages and a support conversation that starts with "the system."

Silent success. The job returns because an exception was swallowed. The queue looks healthy. The user does not.

Clock drift and delayed jobs that fire after a cancellation. If you do not re-read eligibility, you will do favours for ghosts.

Operational consequences

Once people trust the queue, they will put more of the business in it. That is fine if stuck work is visible. If the only view is a log file, you have built a basement.

On-call quality follows this. A retry storm looks like a vendor outage and may be your own uniqueness bug. If you cannot tell the difference quickly, you will restart workers as a personality trait.

There is also a product consequence. Once users learn that "processing" sometimes means "lost," they will tap again. You will then need the idempotency you skipped, except now you need it under load and under support pressure. Design the failure while the tap is still rare.

Lessons

A queue moves time. It does not create safety. Safety is a theory of failure that the code is allowed to execute.

If you cannot say what a second attempt means, you are not ready to have a first attempt out of the user's sight.

What I would do differently today

I would have added an intent row before I added the third retry. I used to treat persistence of "what we meant to do" as extra. It is the whole plot.

I would also have stopped copying tries => 3 from the last job. Defaults are how unlike work becomes one policy.

I would have named the user-facing expiry in the same ticket as the job. If the UI says processing, that sentence needs a clock. Without the clock, support invents one, and their clock will be kinder than the queue deserves.

Closing thought

Dispatching a job is easy because the framework wants to help. Designing the failure is the work. Every retry says what you think the last failure was. If you have not chosen that meaning, the counter will choose it for you, and the counter does not know your domain.

Context

The systems I care about here sit between a product people pay through and a finance function that has to produce invoices, refunds, and a record that survives audit. In a college, that money is tuition, deposits, extras, sometimes partial payments, sometimes a transfer that arrives days after the student thinks they are done. In earlier work it was marketplace and travel money, with local methods that did not behave like a card charge.

The technical shape is familiar. Laravel owns the customer-facing payment. A provider — Stripe in some flows, a bank or specialist transfer product in others — owns the movement. QuickBooks, or something like it, owns the books. Each of these systems is good at its job. None of them agreed, out of the box, on what a "payment" is.

The users of the failure were not only developers. Admissions wanted to know if someone was clear to proceed. Finance wanted to know if the invoice was settled, in which currency, against which item, and whether a fee had been taken out before the money arrived. Support wanted to know what to say when all three screens disagreed. Engineering wanted the ticket to leave the board.

The real problem

The apparent problem was mapping. Map a successful charge to an invoice. Sync the invoice to accounting. Done. That is the diagram that gets approved.

The deeper problem was that success is not one event. There is authorisation, capture, payout, settlement, refund, partial refund, chargeback, a transfer that is pending at the bank, a payment that succeeded in the provider and failed to attach to the right person in your database, a fee that exists only on the provider statement. If your domain model has a boolean called paid, you have already lost a large part of the month-end conversation.

There is a second deeper problem, and it is organisational. Engineering tends to optimise for the customer path: can they complete checkout. Finance optimises for the period: can we explain cash. Those are different success conditions. A checkout that writes paid at the moment the provider says okay will make the product feel snappy. It will also create rows that finance cannot journal because the money is not yet theirs, or is theirs minus a fee nobody modelled, or is theirs but against the wrong invoice because the student paid a balance that spanned two items.

I have watched technically neat integrations create operational work because they stored a provider status and assumed the rest of the company would learn to speak provider. They will not. They should not have to. If your admin screen is a thin mirror of Stripe or a bank portal, you have not integrated payments. You have outsourced the explanation.

Constraints

We could not pause collections while we redesigned money. People were paying. Offers had dates. That alone kills any plan that starts with "stop taking payments for a month and do it properly."

Provider behaviour was not ours to tidy. Settlement timing differs. Local transfer products post in ways card APIs do not. Webhooks arrive twice, late, or in an order you did not write a test for. QuickBooks has its own objects, rate limits, and opinions about customers and line items. If your design requires those systems to become simple, the design is a wish.

Historical data already existed. Old payments had been recorded with whatever honesty the previous code could afford. Some were notes. Some were imported. Some were correct in one system and absent in another. A new model that only worked on clean future rows would split the universe into "payments we understand" and "payments we smile about."

The team was small, which meant the same people who shipped checkout were the people finance would ping when a row looked wrong. There was no payments platform team in another building. If the explanation was bad, we would be the ones in the spreadsheet.

And we could not invent precision we did not have. I will not pretend we had a measured reconciliation rate or a clean before-and-after. What we had was a monthly texture: too many conversations that started with "it's paid on their side."

Options considered

One option was to keep the product boolean and let finance reconcile in the provider dashboards. Advantage: engineering stays fast, checkout stays simple. Disadvantage: month-end becomes a human integration layer. Risk: the human layer does not scale with volume or with holidays. Maintenance looks cheap in the repository and expensive in the calendar. Business impact: you will eventually get a version of this product that nobody in finance trusts, even when it is right.

A second option was to make the provider the system of record and sync everything else from it. Advantage: one upstream truth for money movement. Disadvantage: your product rules — who the payer is, which invoice, which academic or commercial item — do not live at the provider. Risk: you become fluent in someone else's object model and clumsy in your own. Complexity shifts into mapping tables and repair jobs. When the provider is down or merely late, your whole explanation stack is late.

A third option was to make accounting the system of record. Advantage: finance will like the sentence. Disadvantage: accounting systems are poor places to run product workflow. They are slower, stricter, and less interested in your edge cases. Risk: you start making product decisions in a ledger tool, then invent a second unofficial ledger in the application anyway.

The fourth option was to keep an internal money language — intent, captured, settled, allocated, refunded — and treat every external system as a witness. Advantage: support and finance can be taught one story. Disadvantage: you now have to write that story, persist it, and accept that witnesses disagree in time. Risk: if the internal states are sloppy, you have only added vocabulary. Complexity is real: allocations, partials, fees. Maintenance is higher in code and lower in operations if you finish the job.

Decision

We kept an internal ledger-shaped history, even though we were not a bank and did not need to pretend to be one. Provider events updated that history. Accounting sync read from it. The product UI was allowed to say "paid" only when the meaning was one we could defend to finance, not merely when a charge API had returned 200.

I chose that because the other options all exported the hard part to a person. I had already seen what that person does: they keep a side spreadsheet, and then the spreadsheet becomes the real system. At that point your integration is decoration.

The decision was also a refusal. We refused to mark an invoice settled solely because a webhook arrived. We refused to create accounting objects as a side effect of a controller that was also trying to render a success page. Money that matters to a month-end has to pass through a place that can be replayed.

Why not the provider as source of truth? Because our product knew which person, which invoice, and which promise the money was for. The provider knew that a payment method moved value. Those are different facts. Collapsing them is how you get a successful charge against the wrong life.

Implementation / Process

The useful model was small. A payment attempt. A set of events that happened to it. An allocation onto an invoice or balance. A link to an external id we could search when someone forwarded a screenshot from a portal. Fees as their own lines when we knew them, and an honest unknown when we did not.

Request flow for checkout stayed ordinary: create the attempt, send the user to the provider or collect the instrument, wait. The important work moved off the request. A job consumed the provider event, wrote the internal event if we had not seen it, and only then considered allocation. Another job, slower and more cautious, proposed accounting writes. I do not mean we automated judgement we did not have. I mean we stopped mixing "talk to Stripe" and "talk to QuickBooks" in the same nervous function.

Idempotency sat on external event ids and on our own allocation keys. That is dull to say and expensive to skip. Partial success toward accounting is how you get duplicate invoices and a finance team that stops believing your sync.

We also changed the admin language. Screens showed the internal state first, then the provider state as evidence. That sounds cosmetic. It trains everyone, including us, not to treat the provider as the product.

Monitoring was less about throughput and more about stuckness: attempts that aged in a pending state, events we could not parse, accounting writes that failed after money had already moved. Those are the queues that become month-end.

Problems and failures

We assumed settlement language would be teachable in a single meeting. It was not. People had been trained by the old boolean. For a while, the richer screen created more tickets, not fewer, because it told the truth and the truth was uglier than "paid."

Partial payments were harder than refunds. A refund is at least a story people know. A student or customer who pays part of an invoice, then another part by a different method, then gets a discount applied by staff, produces a history that will humiliate a naive unique constraint. We found that only when a real person did it, which is when they always do it.

Fees embarrassed us. Some providers make the fee obvious. Others bury it in a payout. If you allocate the gross to an invoice and the bank receives the net, finance will ask where the rest went. If you allocate the net, the customer receipt looks wrong. We had to admit the fee as a first-class thing instead of hoping it would wash in a batch.

The accounting sync lagged in ways that looked like bugs and were actually honesty. We waited until we had an allocation we trusted. Finance, reasonably, wanted the invoice in QuickBooks earlier. That tension never fully disappears. If you sync too early, you repair. If you sync too late, you are accused of not syncing.

There was also a communication failure I still dislike. Engineering talked about states. Finance talked about periods and accounts. For too long we used the same English words for different objects. A shared glossary would have been worth more than another retry policy.

Trade-offs

We made checkout slightly less triumphant. The user-facing success could still be warm. Internally we accepted a window where money was received but not yet allocated. That is worse for a dashboard that wants all green. It is better for a month that has to close.

We spent engineering time on a model that does not impress in a sprint review. You cannot screenshot an allocation table and get the same reaction as a new payment method logo. I still think that trade is correct if the organisation has a finance function that is not optional.

We did not get perfect consistency across systems. We got a defined delay and a defined owner. Anyone who promises instant agreement between a bank, a card platform, and an accounting API is selling a diagram.

Result

The conversations changed shape. They did not vanish. Instead of "the provider says paid, why doesn't the invoice?", we could usually point to a missing allocation, a pending settlement, or a fee line that had not been booked. That is a worse sentence for engineering pride and a better sentence for operations.

Finance still used provider portals. They used them as evidence, not as the only map. Support had a screen that was closer to a story and further from a raw JSON dump. I count that as the integration starting to exist.

I will not invent a closing-time improvement. What I can say is that the month stopped depending as heavily on one person who "knew how to fix the file."

What I would do differently today

I would have brought finance into the state machine before the first provider went live, not after the first ugly month. We designed a good customer path and then tried to hang accounting on it. That is backwards if the organisation cannot operate without the books.

I would have treated fees and partials as day-one cases, even if the first method was a simple card charge. The first method teaches the model. If the model cannot hold a partial, the second method will arrive and break the month.

I would also have refused any admin UI that displayed provider enums as the primary status. It trains the whole company to become junior operators of someone else's product. That is not leverage. It is a slow headcount tax.

Broader lesson

A payment integration can be technically successful and still be an operational failure. The test is not "did the API return success." The test is "can a person who does not write code explain the same money in the product, at the provider, and in the books, without inventing a fourth story in a spreadsheet."

If you cannot pass that test, you do not have a payments platform. You have a checkout.

Closing thought

Money is not moved when the charge succeeds. Money is moved when the organisation can defend what happened. If your integration cannot survive that sentence, the next month will keep finding you, no matter how clean the webhook handler looks.

Context

These systems sit under people who do not think in resources. They think in terms, invoices, attendance, visas, rooms, flights, agent commissions. The software is asked to be the place those nouns stop being rumours.

The users are staff who will be there after you ship. They will invent exceptions because the official process never covered Tuesday. They will also be asked for reports that assume the official process was followed. That contradiction is the product.

I have also seen marketplace and multi-tenant back-offices with the same shape: many clients, one codebase, a lot of "just this customer." The label ERP is less important than the load. The load is operational memory.

Finance and compliance make this sharper, but they are not required for the pattern. Attendance, accommodation, agent bookings — anything that will be argued about next month — belongs in the same family. If a manager will ask "what was true then," you are in ERP territory even if nobody uses the word.

The problem

The apparent problem is missing features. We need another module. Another form. Another export.

The real problem is that the business is not a set of current rows. It is a set of rules that change, plus a history that must remain explainable after the rules change. A student's status last March still matters in September. A booking that was cancelled still matters to a refund. A price that was wrong and then corrected still matters to an audit.

Ordinary CRUD is allergic to that. It wants the latest name, the latest flag, the latest total. Operations want to know what was true when a decision was taken.

That allergy shows up in the smallest designs. A student name change that overwrites the name on a letter already sent. A price edit that changes a booking already invoiced. A status that means "current" and also means "as of the last save," which are not the same sentence. CRUD invites you to believe the row is the world. In an operational system the row is a view of a story, and the story has already been told to someone else.

The tempting solution

Model the nouns you can see. Users, orders, invoices, students, rooms. Give them create-read-update-delete. Add a status enum when someone complains. Add a notes field when the enum fails. Add an admin who can edit anything when the notes field fails.

This produces software that looks complete in a demo. You can click through a happy life. The unhappy life — the one that pays the salaries of the people in the office — lives in the notes field.

Why that is not enough

Business rules in these systems are rarely predicates on a single row. They are sequences. You may not invoice until X. You may not change Y after Z has been sent. You may change Y if a named role records a reason. That is not a form validation. That is a process.

Permissions follow the process, not the table. The same person may create a record and not be allowed to reverse the financial consequence of it. If your authorisation model is "can update invoices," you have already lost the plot.

Reporting wants shapes the transactional model does not enjoy. Month-end, attendance windows, agent statements. If you only have the live row, you will rewrite history every time you "fix" a field.

Integrations then attach to whatever you called the current truth. Accounting, email, a mobile app. They amplify whatever sloppiness you hid in CRUD.

There is also a staffing consequence. CRUD systems train developers to think in resources. Then a request arrives that is actually a period close, or a status that must not move after a document was issued, and the team reaches for another column. The column works for a week. The report breaks in a month. The organisation learns that engineering "does not understand the business." What they mean is that we modelled a form.

I have more sympathy for that accusation than I used to. If your schema cannot say what was true in a previous window, you will keep asking operations to be the memory. They will, and they will stop trusting the screens.

Options

Stay CRUD and let staff keep a spreadsheet for the real process. Advantage: the application stays simple. Disadvantage: the spreadsheet becomes the ERP, and your software becomes a data-entry tax.

Buy a generic ERP and customise it until it resembles the organisation. Advantage: you inherit a vendor's gravity. Disadvantage: you inherit their nouns, and the customisation becomes a second product you do not fully own. Sometimes still correct. Not a way to avoid understanding the domain.

Build a domain model around transitions, documents, and periods, even if the UI still looks like forms. Advantage: the software can explain itself. Disadvantage: slower to the first screen. You will be accused of over-engineering by anyone who has only seen the demo path.

Event-source everything. Advantage: history is native. Disadvantage: you have bought a research project. I want history where money, status, and compliance live. I do not want an event log for every typo in an address.

Trade-offs

You trade the pleasure of a thin resource API for the burden of saying what a change means. That burden is the job.

You also trade some user freedom. A good operational system refuses edits that would make the past unintelligible. Staff will call that rigidity. The alternative is a system that agrees with whoever last saved.

You should not trade away a way to record an exception. Exceptions will happen. If they cannot be recorded, they will be hidden.

Decision

I design these products as records of decisions, not as bags of fields. A status change is an event with an actor. A financial figure that has been used downstream is not a cell you casually overwrite. A report has a period and a definition, not "whatever the table says tonight."

I still use Laravel and MySQL. This is not a stack speech. It is a refusal to let the framework's resource generators decide the domain.

I also refuse to let the admin user be the process. If the only way to handle Tuesday is "get an admin to edit the row," you have not built an ERP. You have built a database with a dress on. The admin path can exist. It should be a recorded exception, not the weekday.

Implementation / Thinking process

I look for the facts that other departments already treat as evidence: money, attendance, enrolment, fulfilment, anything that has been emailed or invoiced. Those get transitions and reasons. The rest can often stay as ordinary records until they graduate into evidence.

Permissions are named after work, not tables. May issue this letter. May reverse this allocation. May open this period. Table-level CRUD maps poorly onto that and trains you to hand out "update" like a master key.

When a requirement is "we need to change this after the fact," I ask who will read the old value next month. If the answer is finance, auditors, or a student who received a document, we are talking about a new event, not an edit.

I keep a place for the unofficial. A recorded exception with an owner is uglier than a clean model and more honest than a notes column that means everything.

Periods matter. A lot of operational truth is about a term, a month, a booking window. If you only store current flags, you will reconstruct periods from timestamps and get them wrong. I would rather a first-class period on the facts that are reported than a clever query that infers the academic year from created_at.

Permissions, again, follow work. The person who can enter an attendance mark may not be the person who can reopen the window. That is not fussiness. That is how you stop a cleanup from becoming a rewrite of history.

Failure modes

The model stays clean and the office invents parallel books. You will only notice when a report disagrees with a person who is believed.

Someone is given global edit because "they need to fix things." Global edit is how history dies.

A migration or a "data cleanup" rewrites statuses without events. The software becomes confident and wrong.

A new module is built as CRUD because the team is tired, and it becomes the next place exceptions hide. ERP rot is usually local, then contagious.

Operational consequences

Staff trust is binary in these systems. They either use the screen as the story or they use it as a form to satisfy someone else. You can tell which you have by whether they still keep a private file.

Once the system is the story, change gets slower and more valuable. That is not failure. That is the product succeeding at being the record. People who only measure feature count will hate this phase. They are measuring the wrong thing.

The other consequence is political. Once the system is the record, changing a status meaning is a change to how the organisation remembers itself. That is slower than a rename in a CRUD app, and it should be. If you want speed here, you are asking to rewrite history without a meeting.

Lessons

If the organisation would be in trouble if the software forgot last term, you are not building CRUD. You are building memory with rules.

Memory without rules is a junk drawer. Rules without memory are a brochure.

What I would do differently today

I would have named "evidence facts" in the first month of a back-office, instead of discovering them when a report and a human disagreed. The list is usually short. It should be sacred.

I would have been harsher about notes fields. A note is allowed to explain. It is not allowed to be the only place a rule exists.

I would also have drawn the document boundary earlier. A letter, an invoice, an attendance register — anything that left the building — is not a row you tidy later. It is a published fact. Treating published facts as editable fields is how you get a clean demo and an unexplainable September.

Closing thought

An ERP, or whatever you call the operational core, is not a database with forms. It is a record of how the business is allowed to move. Design for the movement and the memory. The forms are how people enter the room, not what the room is for.

Context

I was working in products that had already outlived their first design. Multi-tenant back-office systems, school administration, later college operations: the kind of software that sits under enrolment, invoices, staff routines, and a pile of exports that finance or operations still open in a spreadsheet. These are not greenfield APIs. They are systems with years of exceptions encoded as if they were features.

The teams around them were small. That matters. A small team can keep a messy system alive for a long time because two or three people still remember why a column is named the way it is, or why a job runs at 02:10 instead of on a clean hourly cadence. The product looks cheaper to own than it is. The cost is stored in people's heads.

The architecture was the ordinary kind that accumulates around Laravel and MySQL. One database. Shared tables that several modules treated as their own. Helpers that started as convenience and became the only place a rule still existed. A few queued jobs, some of them doing more than their names suggested. Frontend screens that talked to controllers which talked to models which talked to more models. Nothing exotic. That was part of the problem. Exotic systems get respect. Ordinary systems get one more feature.

The real problem

The surface problem was always the same: this change will take longer than it should. Product would hear that as engineering being cautious, or slow, or precious about cleanup. Engineering would hear it as product not understanding the codebase. Both readings are too flattering. The deeper problem was that the system no longer had a trustworthy map of cause and effect.

When a status lives in one table, a derived flag in another, a notification template that keys off a string, and a report that hard-codes the old values, you do not have a status field. You have a rumour that several subsystems overheard at different times. Changing the rumour is not a migration. It is a search problem.

I have also seen the inverse, which looks healthier and is not. The code is tidy enough. Naming is fine. Tests exist around the happy path. And still, a change is expensive because the important rules are not in the code. They are in a conversation from 2019, or in a customer's private exception, or in a finance process that was never written down because everyone assumed the application already knew it. The application did not know it. A person did, and that person used the admin screen as a workaround.

So the real problem is not "legacy syntax" or "old Laravel." Those are symptoms you can schedule. The real problem is lost coupling. Some of it is technical: foreign keys that were never declared, observers that fire on the wrong events, caches that nobody invalidates on the write you are about to add. Some of it is organisational: the last person who understood a workflow has already moved on, and the software is now the only remaining witness, except the software is incomplete.

Constraints

We could not stop the product. That is the constraint that makes this whole subject honest. Students, agents, staff, and finance do not pause because engineering wants a cleaner module boundary. If your plan requires a quiet quarter, you do not have a plan. You have a wish.

We also could not hire our way out of it on a useful timeline. A small team can add one person. It cannot add a second product org. That means any strategy that depends on a parallel rewrite team, or on freezing features for six months, was politically and operationally false even if it was technically tidy.

Data was already dirty in the way real data is dirty. Historical rows did not obey the rules we wanted to enforce going forward. Some tenants or departments had been given exceptions that were never modelled. Any attempt to "just add a constraint" would fail first on the past, then on the argument about whether the past should be rewritten.

Third-party edges made it worse. Payments, email, accounting, sometimes a mobile client that would not be updated on our schedule. A change that looked local could become an integration change, and integration changes have calendars that do not belong to you.

The last constraint was fear, and I do not use that word loosely. Once a team has been burned by a small change that cascaded, they start padding every estimate and avoiding the centre of the system. That fear is rational. It is also how a changeable system finishes the job of becoming unchangeable.

Options considered

The first option was to keep shipping on top and treat the friction as normal. Advantage: no ceremony, no argument with stakeholders, no migration risk this sprint. Disadvantage: every subsequent change inherits a slightly worse map. Risk: the next "small" request hits a coupling you cannot see, and you spend the week in production archaeology. Maintenance impact: the codebase stays familiar to the people who already know it and becomes hostile to everyone else. Business impact: you look fast until you suddenly look unreliable.

The second option was a rewrite. Same product, new structure, migrate later. Advantage: you get to draw the boundaries you wish you had. Disadvantage: you now own two systems, plus the gap between them, plus every feature request that arrives while you are busy being principled. Risk: the rewrite learns the documented rules and misses the ones staff perform by hand. I had already seen enough of that pattern in older products to distrust the confidence it produces in the first month. Complexity is highest exactly where the business is least patient: parallel behaviour, dual writes, "temporary" flags that become permanent.

The third option was local cleanup only when a change forced us into a module. Advantage: the business keeps moving, and the cleanup has a customer attached to it. Disadvantage: you only improve the rooms you are already standing in. Whole corridors stay dark. Risk: you congratulate yourselves for extracting a service class while the shared table underneath still belongs to four domains. Maintenance gets better in patches and stays bad in the seams.

The fourth option, the one I kept coming back to, was to treat changeability as a product of knowledge, not of folders. Before choosing rewrite or refactor, make the couplings visible. Write down the real workflow. Trace the writes. List the jobs and reports that read the same facts. Then change the smallest thing that makes the next change safer. That option is slower to explain in a meeting. It is the only one that does not pretend the system is simpler than the organisation using it.

Decision

We did not rewrite. We also did not accept "just ship it" as a strategy. The decision was to refuse changes that entered a tangled area without leaving a clearer map behind. Not a grand platform. A rule: if you touch a shared fact, you have to say who else consumes it, and you have to make that consumption harder to miss next time.

I chose that because the rewrite fantasy failed the constraints immediately, and the do-nothing option was already failing in slow motion. I had enough scars from older multi-tenant work to know that a clean new schema does not create a clean business. It creates a second place for the same exceptions to hide.

The "why" was not elegance. It was survival of the estimate. I wanted a world where a status change had a list of dependents we could read, instead of a list we discovered in staging. That is a modest ambition. It is also the difference between a lead who can keep promises and a lead who keeps apologising.

Implementation / Process

The work looked unglamorous. We started with the facts that several parts of the product already argued about: application state, payment state, a person's current status in a process, whether a record was safe to email about. For each of those, we wrote a short inventory. Not a wiki novel. A list: tables, jobs, UI screens, exports, and the one or two people who still understood the exceptions.

Then we made the write path narrower. Instead of letting controllers and random observers mutate the same columns, we pushed those writes through one place that could log the transition and, later, emit a single event. I am not describing event sourcing. I am describing a door. If there is one door, you can stand in front of it.

We also stopped treating reports as innocent readers. A report that re-implements a business rule is a second system. Where we could, we made the report read the same derived values the UI used. Where we could not, we marked the report as a known liar with an owner. That sounds rude. It is kinder than letting finance discover the lie during a busy week.

None of this required a new infrastructure story. MySQL stayed. Redis stayed for the things it was already doing. The deployment process stayed boring on purpose. The change was in how we were allowed to touch shared state, and in the expectation that a pull request in a dangerous area included the inventory, not just the feature.

Problems and failures

The first failure was assuming the inventory would be obvious from the code. It was not. Code review finds functions. It does not find the operations person who changes a record by hand every Thursday because the official flow never covered their case. We learned that only by sitting with the workflow instead of inferring it from models.

The second failure was cleanup enthusiasm. Once you see a shared table, you want to split it. We tried to move too much meaning in one change and created a transition period where old and new columns disagreed. The application still ran. A couple of screens told different stories. That is worse than an exception. Users trust screens.

We also under-estimated how many background jobs were quietly rewriting state to "fix" yesterday's incompleteness. Those jobs had been compensating for missing rules. When we tightened the write path, the jobs started failing or no-op'ing, and for a while it looked like the new discipline had broken processing. It had revealed that processing was a pile of apologies.

Communication lagged the technical work. Product still heard "this is bigger than it looks" without seeing the map. Until we started showing the dependent list in planning, the constraint sounded like mood. After we showed it, the conversation got more adult and more tense. Both are preferable to fake agreement.

Trade-offs

We traded speed of the next local feature for speed of the next honest estimate. That is not a slogan. Some tickets that used to be "quick" became a day longer because the inventory was now required. I think that is the correct tax. The alternative tax is paid later, in production, with worse interest.

We also traded a certain kind of individual heroism. The engineer who "just knows" where to patch things is useful until they become the system. Making couplings visible reduces that person's mystique and increases the team's ability to survive their holiday. Not everyone enjoys that trade at first.

What we did not buy was a beautiful architecture. The monolith was still a monolith. Shared tables still existed. We accepted inconsistency at the edges so we could make the centre less mythical. Purity would have been a rewrite. We did not have the right to that fantasy.

Result

The useful result was not a metric. It was a change in the quality of surprise. We still found missed dependents. We found fewer of them after deploy, and more of them while the change was still a branch. Planning conversations started referring to specific jobs and exports instead of a vague sense that the module was "spicy."

New people could be shown a write path instead of being told to be careful. That sounds like a small cultural shift. In a small team it is the whole game. Care does not scale. Paths do.

The product did not become easy. It became slightly more honest about where it was difficult. I will take that over a demo of a future system that does not have to answer Monday's tickets.

What I would do differently today

I would start the inventory with operations, not with the schema. I went to the tables first because I am a programmer and tables feel like truth. They are residue. The truth is the workflow someone can perform with their eyes closed. If I had sat in those workflows earlier, I would have found the compensating jobs and the handwritten exceptions before I tightened the door.

I would also have been stricter about reports. I treated a few of them as politically untouchable and let them keep their private logic. That was cowardice dressed as pragmatism. A report that contradicts the application will eventually set the application's reputation, not the other way around.

And I would have written the "who consumes this fact" list into the code, not only into a document. Documents rot on the same schedule as good intentions. A test that fails when a new reader appears, or a single module that is allowed to import a state transition, lasts longer than a markdown file nobody opens during review.

Broader lesson

Software becomes hard to change when the organisation can no longer see the consequences of a write. Frameworks, versions, and hosting are downstream of that. If you can name the facts that several departments depend on, and you can name the paths that change those facts, you still have a system you can steer. If you cannot, you have a runtime and a set of superstitions.

This is why I am now suspicious of health checks that only talk about availability. Availability is the easy virtue. Changeability is the one that decides whether the next year of product work is compounding or circling.

Closing thought

A system that still serves traffic can already be too expensive to evolve. If you want to know which one you have, do not start with the uptime graph. Start with a change that should have been small, and notice how many rooms you have to walk through before you are allowed to touch the obvious field.

Context

Operational products collect these requests because the organisation is already making the judgement by hand. Someone emails. Someone writes a note. Someone changes a status they were not supposed to change, then tells a colleague. The software is asked to "add an approval" when that informal gate has become too risky or too tiring to leave in chat.

The team hearing the request is usually small. That matters. There is no analyst whose only job is to expand the sentence. If engineering accepts the boolean, the boolean ships. The real rules arrive later as exceptions, and exceptions are how you grow a second system beside the first.

I have sat on both sides of this. As the person who wanted to finish the ticket. As the person who later had to explain why an approved record could not be undone, or why it was undone too easily.

The word shows up everywhere: refunds, letters, discounts, room changes, a status that unlocks the next office. If you treat each as a new flag, you will have six booleans and still no absence path. The questions repeat. That is a clue that you are looking at a type of work, not a one-off control.

The problem

The stated problem is missing control. Anyone can complete the action. We need a responsible person to say yes.

The actual problem is that "approval" is not one fact. It is a set of questions the organisation has usually not answered in the same room. Who is allowed to approve this class of thing. What happens if they are absent. Can two people approve, and if they disagree, which one wins. Can approval be reversed after a side effect — a payment, an email, a status another department already used. Do historical records need to look approved even if the rule did not exist then. What should an administrator be allowed to bypass, and how is that bypass recorded.

If you skip those questions, you do not skip the work. You schedule it for production, where a staff member will invent the answer with a workaround.

The tempting solution

Most of us reach for a flag. approved_at, approved_by, a policy that hides the next button. Maybe a notification. It demos well. The stakeholder can see the gate. The pull request looks like a feature.

If you are slightly more careful, you add a role check. Only finance. Only admissions. Only the assigned manager. That still fits in an afternoon if you do not ask what "assigned" means when the assignee has left.

Why that is not enough

A boolean cannot represent absence. The approver is on leave. The work is stuck, so someone with a broader admin role completes it, and now you have taught the organisation that the gate is optional for people who know where the admin screen is.

A boolean cannot represent reversal cleanly. If the side effect has already left the building — an invoice, a message, a downstream status — "unapprove" is not the inverse of a tick. It is a new business event with its own audience.

A boolean cannot represent history. Old rows have no approver. Reports will lie or fail. Someone will ask you to backfill. Backfill without a rule is fiction.

And a boolean cannot represent the thing that usually matters most: the action after approval. Approval is rarely the product. It is a door. If you model the door and not the room, you will approve things into a state the rest of the system does not understand.

Options

Keep the informal process and refuse the ticket. Advantage: you do not encode a half-rule. Disadvantage: the organisation already asked for software because the informal process is failing. Sometimes the honest answer is still no, until they can name the rule.

Ship the boolean and promise to "harden it later." Advantage: speed. Disadvantage: later never has a stakeholder. The boolean becomes the contract.

Model a small state machine: requested, approved, rejected, withdrawn, superseded, with actors and reasons, and treat side effects as listeners of a transition, not of a page submit. Advantage: you can talk about absence and reversal as states. Disadvantage: more to build and to teach. Risk: you over-model a one-person tick that really is a tick.

Put the workflow in a generic engine. Advantage: flexibility. Disadvantage: you have invented a programming language for staff, and the next change becomes configuration archaeology. I am wary of this unless the organisation already lives in workflow tools.

Trade-offs

A real approval model is slower than a flag and more honest than a meeting. You spend time in questions that feel like they are not engineering. They are the engineering. The code is the cheap part.

You also give up a certain kind of flexibility. Once states exist, people will ask for a new one every term. That is the cost of making the process visible. Invisible process is flexible because nobody can audit it.

What you should not trade away is history. If the record cannot say who decided, when, and whether a later reversal happened, you have built a decoration.

Decision

I treat "add an approval" as a design conversation with a written outcome before a migration. The outcome has to name the actor, the absence path, the reversibility rule, and what is allowed to happen after the transition. If those cannot be named, the ticket is not ready. That is a more useful no than "we are busy."

When they can be named, I want an explicit transition, not a lone boolean, even if the first version only has two states. A transition can grow a reason and a second actor. A boolean can only grow regret.

If the organisation cannot name the approver, I ask who they phone today. That person is the actor, even if the org chart says otherwise. Software that implements the chart and not the phone tree will be walked around by the phone tree.

Implementation / Thinking process

I start on paper, not in a migration. List the happy path. List the away path. List the "we were wrong" path. List the records that already exist. If any of those is a shrug, stop.

In the application, the write that matters is the transition. Controllers may collect the click. They should not be the only place the rule lives. Jobs and admin tools will otherwise invent a second door.

Side effects wait for the transition they actually depend on. If an email should not go until approval, the email listens to approved, not to created. If a payment already happened, approval cannot pretend to be a precondition you can insert after the fact without a separate reconciliation story.

Permissions follow the workflow, not the other way around. "Is admin" is not a substitute for "may approve this class of record." Admin bypass, if it exists, is a recorded event with a reason. Otherwise you have two products: the official one and the one people with the wider role use on Fridays.

Notifications are part of the model. An approval that nobody hears about is a stuck state with a UI. I do not need a real-time stack for this. I need a reliable "you have work" that does not share a queue with a bulk campaign.

Old records need a policy in writing: grandfather as unapproved-but-grandfathered, or leave them outside the new reports, or run a one-time review. What I do not want is a silent UPDATE that paints the past. That is fiction with a timestamp.

Failure modes

The most common failure is the holiday. The only approver is away, the work is urgent, and someone "just this once" uses a power you left lying around. You will not hear about it as a bug. You will hear about it as a process.

The second is double approval without a rule. Two people click. The system accepts both or rejects the second opaquely. Staff lose trust in the screen.

The third is rewriting the past. A report needs every historical row to look compliant, so someone updates old records to approved with a null actor. That is how you destroy the only evidence you had.

The fourth is notifying the wrong person, or nobody, and concluding that "approvals do not work" when the model is fine and the attention path is not.

Operational consequences

Once the gate exists, the organisation will route work to it. Queues will form. Someone will ask for a dashboard of pending approvals. That dashboard is not scope creep. It is the product you implied when you accepted the word.

Support will also start speaking in the new language. "It is waiting for approval" becomes a status they tell students or customers. If your states are sloppy, that sentence will be a lie, and you will have exported the lie.

There is a reporting consequence as well. Once the word exists, someone will ask for a count of approved versus pending, then for a time-to-approve, then for a list of people who sit on work. Those numbers will be used in meetings. If your model cannot distinguish "waiting for a person" from "waiting for a side effect," you will publish a metric that punishes the wrong office. The boolean cannot save you there. The states can, if you named them.

Lessons

Product language compresses domain. Engineering's job is to decompress it in public, early, while the cost of a question is a meeting and not a data repair.

A flag is a valid implementation of a rule that has been fully stated. It is a poor implementation of a rule that has not.

What I would do differently today

I would refuse to estimate in hours until the four questions have answers: who, absence, reversal, history. I used to estimate the button and then discover the state machine in review. That wastes the only cheap moment.

I would also write the absence path into the first version even if it is ugly — a named deputy, an expiry, a recorded admin override. Shipping a gate with no away path is how you train people to walk around it.

Closing thought

When someone asks you to add an approval, they are asking you to encode a piece of how the organisation accepts risk. If you only add a boolean, you have encoded a screenshot. The domain will still happen. It will happen in the gaps.