Engineering Journal · ENGINEERING · June 21, 2023 · 16 min read
The System Gets Hard to Change Long Before It Gets Hard to Run
Most production systems still serve traffic. The real decay is quieter: every change starts needing archaeology, because knowledge and coupling have already escaped the code.
By Golam Sorwar, Tech Lead and Full Stack Engineer in Dublin.
I used to judge a codebase by whether it was up. If users could log in, payments could settle, and the error rate looked ordinary, I treated the system as healthy. That was a useful instinct when I was newer. It is a dangerous one once you are the person asked to change something that already has a life around it.
The first time I really noticed the gap was not during an outage. It was during a request that sounded small. A new status. An extra field on an existing form. A report that already existed, just with one more filter. Nobody thought we were asking for architecture. Then the estimate grew, not because the feature was ambitious, but because nobody could say with confidence which other screens, jobs, and exports would shift if we touched the obvious table.
That is the moment I started taking seriously: a system can keep running and still be close to unchangeable. Uptime hides that. Dashboards hide that. The people who live in the product every day feel it first, usually as hesitation. They stop asking for the change they actually need and start asking for the change they think engineering will survive.
Context
I was working in products that had already outlived their first design. Multi-tenant back-office systems, school administration, later college operations: the kind of software that sits under enrolment, invoices, staff routines, and a pile of exports that finance or operations still open in a spreadsheet. These are not greenfield APIs. They are systems with years of exceptions encoded as if they were features.
The teams around them were small. That matters. A small team can keep a messy system alive for a long time because two or three people still remember why a column is named the way it is, or why a job runs at 02:10 instead of on a clean hourly cadence. The product looks cheaper to own than it is. The cost is stored in people's heads.
The architecture was the ordinary kind that accumulates around Laravel and MySQL. One database. Shared tables that several modules treated as their own. Helpers that started as convenience and became the only place a rule still existed. A few queued jobs, some of them doing more than their names suggested. Frontend screens that talked to controllers which talked to models which talked to more models. Nothing exotic. That was part of the problem. Exotic systems get respect. Ordinary systems get one more feature.
The real problem
The surface problem was always the same: this change will take longer than it should. Product would hear that as engineering being cautious, or slow, or precious about cleanup. Engineering would hear it as product not understanding the codebase. Both readings are too flattering. The deeper problem was that the system no longer had a trustworthy map of cause and effect.
When a status lives in one table, a derived flag in another, a notification template that keys off a string, and a report that hard-codes the old values, you do not have a status field. You have a rumour that several subsystems overheard at different times. Changing the rumour is not a migration. It is a search problem.
I have also seen the inverse, which looks healthier and is not. The code is tidy enough. Naming is fine. Tests exist around the happy path. And still, a change is expensive because the important rules are not in the code. They are in a conversation from 2019, or in a customer's private exception, or in a finance process that was never written down because everyone assumed the application already knew it. The application did not know it. A person did, and that person used the admin screen as a workaround.
So the real problem is not "legacy syntax" or "old Laravel." Those are symptoms you can schedule. The real problem is lost coupling. Some of it is technical: foreign keys that were never declared, observers that fire on the wrong events, caches that nobody invalidates on the write you are about to add. Some of it is organisational: the last person who understood a workflow has already moved on, and the software is now the only remaining witness, except the software is incomplete.
Constraints
We could not stop the product. That is the constraint that makes this whole subject honest. Students, agents, staff, and finance do not pause because engineering wants a cleaner module boundary. If your plan requires a quiet quarter, you do not have a plan. You have a wish.
We also could not hire our way out of it on a useful timeline. A small team can add one person. It cannot add a second product org. That means any strategy that depends on a parallel rewrite team, or on freezing features for six months, was politically and operationally false even if it was technically tidy.
Data was already dirty in the way real data is dirty. Historical rows did not obey the rules we wanted to enforce going forward. Some tenants or departments had been given exceptions that were never modelled. Any attempt to "just add a constraint" would fail first on the past, then on the argument about whether the past should be rewritten.
Third-party edges made it worse. Payments, email, accounting, sometimes a mobile client that would not be updated on our schedule. A change that looked local could become an integration change, and integration changes have calendars that do not belong to you.
The last constraint was fear, and I do not use that word loosely. Once a team has been burned by a small change that cascaded, they start padding every estimate and avoiding the centre of the system. That fear is rational. It is also how a changeable system finishes the job of becoming unchangeable.
Options considered
The first option was to keep shipping on top and treat the friction as normal. Advantage: no ceremony, no argument with stakeholders, no migration risk this sprint. Disadvantage: every subsequent change inherits a slightly worse map. Risk: the next "small" request hits a coupling you cannot see, and you spend the week in production archaeology. Maintenance impact: the codebase stays familiar to the people who already know it and becomes hostile to everyone else. Business impact: you look fast until you suddenly look unreliable.
The second option was a rewrite. Same product, new structure, migrate later. Advantage: you get to draw the boundaries you wish you had. Disadvantage: you now own two systems, plus the gap between them, plus every feature request that arrives while you are busy being principled. Risk: the rewrite learns the documented rules and misses the ones staff perform by hand. I had already seen enough of that pattern in older products to distrust the confidence it produces in the first month. Complexity is highest exactly where the business is least patient: parallel behaviour, dual writes, "temporary" flags that become permanent.
The third option was local cleanup only when a change forced us into a module. Advantage: the business keeps moving, and the cleanup has a customer attached to it. Disadvantage: you only improve the rooms you are already standing in. Whole corridors stay dark. Risk: you congratulate yourselves for extracting a service class while the shared table underneath still belongs to four domains. Maintenance gets better in patches and stays bad in the seams.
The fourth option, the one I kept coming back to, was to treat changeability as a product of knowledge, not of folders. Before choosing rewrite or refactor, make the couplings visible. Write down the real workflow. Trace the writes. List the jobs and reports that read the same facts. Then change the smallest thing that makes the next change safer. That option is slower to explain in a meeting. It is the only one that does not pretend the system is simpler than the organisation using it.
Decision
We did not rewrite. We also did not accept "just ship it" as a strategy. The decision was to refuse changes that entered a tangled area without leaving a clearer map behind. Not a grand platform. A rule: if you touch a shared fact, you have to say who else consumes it, and you have to make that consumption harder to miss next time.
I chose that because the rewrite fantasy failed the constraints immediately, and the do-nothing option was already failing in slow motion. I had enough scars from older multi-tenant work to know that a clean new schema does not create a clean business. It creates a second place for the same exceptions to hide.
The "why" was not elegance. It was survival of the estimate. I wanted a world where a status change had a list of dependents we could read, instead of a list we discovered in staging. That is a modest ambition. It is also the difference between a lead who can keep promises and a lead who keeps apologising.
Implementation / Process
The work looked unglamorous. We started with the facts that several parts of the product already argued about: application state, payment state, a person's current status in a process, whether a record was safe to email about. For each of those, we wrote a short inventory. Not a wiki novel. A list: tables, jobs, UI screens, exports, and the one or two people who still understood the exceptions.
Then we made the write path narrower. Instead of letting controllers and random observers mutate the same columns, we pushed those writes through one place that could log the transition and, later, emit a single event. I am not describing event sourcing. I am describing a door. If there is one door, you can stand in front of it.
We also stopped treating reports as innocent readers. A report that re-implements a business rule is a second system. Where we could, we made the report read the same derived values the UI used. Where we could not, we marked the report as a known liar with an owner. That sounds rude. It is kinder than letting finance discover the lie during a busy week.
None of this required a new infrastructure story. MySQL stayed. Redis stayed for the things it was already doing. The deployment process stayed boring on purpose. The change was in how we were allowed to touch shared state, and in the expectation that a pull request in a dangerous area included the inventory, not just the feature.
Problems and failures
The first failure was assuming the inventory would be obvious from the code. It was not. Code review finds functions. It does not find the operations person who changes a record by hand every Thursday because the official flow never covered their case. We learned that only by sitting with the workflow instead of inferring it from models.
The second failure was cleanup enthusiasm. Once you see a shared table, you want to split it. We tried to move too much meaning in one change and created a transition period where old and new columns disagreed. The application still ran. A couple of screens told different stories. That is worse than an exception. Users trust screens.
We also under-estimated how many background jobs were quietly rewriting state to "fix" yesterday's incompleteness. Those jobs had been compensating for missing rules. When we tightened the write path, the jobs started failing or no-op'ing, and for a while it looked like the new discipline had broken processing. It had revealed that processing was a pile of apologies.
Communication lagged the technical work. Product still heard "this is bigger than it looks" without seeing the map. Until we started showing the dependent list in planning, the constraint sounded like mood. After we showed it, the conversation got more adult and more tense. Both are preferable to fake agreement.
Trade-offs
We traded speed of the next local feature for speed of the next honest estimate. That is not a slogan. Some tickets that used to be "quick" became a day longer because the inventory was now required. I think that is the correct tax. The alternative tax is paid later, in production, with worse interest.
We also traded a certain kind of individual heroism. The engineer who "just knows" where to patch things is useful until they become the system. Making couplings visible reduces that person's mystique and increases the team's ability to survive their holiday. Not everyone enjoys that trade at first.
What we did not buy was a beautiful architecture. The monolith was still a monolith. Shared tables still existed. We accepted inconsistency at the edges so we could make the centre less mythical. Purity would have been a rewrite. We did not have the right to that fantasy.
Result
The useful result was not a metric. It was a change in the quality of surprise. We still found missed dependents. We found fewer of them after deploy, and more of them while the change was still a branch. Planning conversations started referring to specific jobs and exports instead of a vague sense that the module was "spicy."
New people could be shown a write path instead of being told to be careful. That sounds like a small cultural shift. In a small team it is the whole game. Care does not scale. Paths do.
The product did not become easy. It became slightly more honest about where it was difficult. I will take that over a demo of a future system that does not have to answer Monday's tickets.
What I would do differently today
I would start the inventory with operations, not with the schema. I went to the tables first because I am a programmer and tables feel like truth. They are residue. The truth is the workflow someone can perform with their eyes closed. If I had sat in those workflows earlier, I would have found the compensating jobs and the handwritten exceptions before I tightened the door.
I would also have been stricter about reports. I treated a few of them as politically untouchable and let them keep their private logic. That was cowardice dressed as pragmatism. A report that contradicts the application will eventually set the application's reputation, not the other way around.
And I would have written the "who consumes this fact" list into the code, not only into a document. Documents rot on the same schedule as good intentions. A test that fails when a new reader appears, or a single module that is allowed to import a state transition, lasts longer than a markdown file nobody opens during review.
Broader lesson
Software becomes hard to change when the organisation can no longer see the consequences of a write. Frameworks, versions, and hosting are downstream of that. If you can name the facts that several departments depend on, and you can name the paths that change those facts, you still have a system you can steer. If you cannot, you have a runtime and a set of superstitions.
This is why I am now suspicious of health checks that only talk about availability. Availability is the easy virtue. Changeability is the one that decides whether the next year of product work is compounding or circling.
Closing thought
A system that still serves traffic can already be too expensive to evolve. If you want to know which one you have, do not start with the uptime graph. Start with a change that should have been small, and notice how many rooms you have to walk through before you are allowed to touch the obvious field.