Engineering Journal · INFRASTRUCTURE · January 21, 2025 · 16 min read
Green Checks Are Not the Same as a Safe Release
A pipeline that is green can still be a bad way to change production. Delivery is risk management: what you proved, what you can undo, and who is allowed to be surprised.
By Golam Sorwar, Tech Lead and Full Stack Engineer in Dublin.
I have shipped through pipelines I trusted and pipelines I only obeyed. The difference was not the logo on the badge. It was whether a red run meant something, and whether a green run meant we were allowed to change a live college, a payment path, or a mail worker without a superstition.
People talk about CI/CD as automation. Automation is the easy part. The hard part is deciding what "safe enough to release" means, and building a path that can reverse a decision that looked safe.
A green check that nobody believes is worse than a slow manual deploy. The manual deploy at least has a frightened human.
Context
The work I care about here is PHP applications on familiar hosting — Forge, AWS, Docker — with GitHub Actions or a similar runner, and databases that cannot be treated like cattle. You can replace a container. You cannot casually replace a term of invoices.
Releases also include the unglamorous: migrate, restart workers, clear config, warm something, watch something. If the pipeline only runs tests, you have continuous integration and a folklore production.
I have used GitHub Actions, Bitbucket pipelines, and host-side deploy hooks. The brand does not matter. What matters is whether the same commit produces the same artefact, and whether a human can point at that artefact in production. If those two sentences are soft, you will debug "what is live" as if it were a mystery novel.
The problem
The apparent problem is toil. Deploys are manual. Let us automate them.
The real problem is surprise. Changing production is how you introduce new failure. The organisation wants features and also wants Tuesday to resemble Monday. Those wants fight. Delivery is the process that makes the fight boring.
If your pipeline cannot talk about rollback, migrations, secrets, and environment drift, it is not managing risk. It is typing for you.
The other half of surprise is people. A release that is technically fine can still be a bad Tuesday if it lands in the middle of enrolment, a payment window, or a mail run. Frequency is not automatically virtue. Frequency is virtue when the blast radius is small and the undo is real. A pipeline that can ship twelve times a day and cannot reverse a migration is a machine for making irreversible afternoons cheaper to start.
The tempting solution
On push, install, test, deploy. One workflow. Main is production. Feel like a grown-up.
Add more checks when something burns you, until the pipeline is a museum of past pain and a twenty-minute wait. People start merging because they are tired, not because they are confident.
Why that is not enough
Tests that do not include the dangerous paths — money, tenancy, auth, jobs — produce green that is not about production. I have more patience for a short suite that covers those than a long suite that covers helpers.
Database migrations do not roll back because the workflow has a rollback job. Expanding a column is not the same as contracting it. If you do not have expand/contract, your "undo" is a restore conversation.
Environments lie. Staging that does not run the workers, or does not have the same config shape, will bless deploys that fail in the only environment that matters.
Secrets that live in the runner and in the server and in a chat will eventually disagree. The pipeline cannot save you from three truths.
Feature flags are not a substitute for a broken migration. They are a way to decouple release from launch when the code can tolerate both versions.
There is also the artefact problem. If CI installs dependencies on the runner and production installs them again on the box, you tested a cousin of what you shipped. "It passed" then means "a similar looking tree passed." I want one build. Promote that build. If you cannot, say so, and stop using the green check as a synonym for "what is live."
Options
Manual deploy from a laptop. Advantage: someone is paying attention. Disadvantage: the laptop is the process. Holidays become architecture.
Fully automatic deploy on every green main. Advantage: small diffs. Disadvantage: a flaky test or a compromised token is a production event. I like small diffs. I like a human on the last step when the blast radius is money or mail.
Build once, promote the same artefact, separate the decision to release. Advantage: you know what you tested. Disadvantage: more plumbing. This is the shape I want as soon as "it passed on CI" and "what is on the box" can diverge.
Trunk plus flags versus long-lived release branches. Branches feel safe and rot. I prefer trunk and honesty about flags over a branch that only exists because we are afraid.
Trade-offs
You trade speed of merging for speed of believing. A pipeline that is fast and flaky trains people to retry. A pipeline that is slow and true trains people to batch. I would rather fix flakes than add retries.
You trade some autonomy. A required check is a gate. Gates need owners. An unowned required check becomes a local legend about who knows the YAML.
You should not trade away a dry run for migrations in production-shaped data, even if it is a subset. The first time you meet a trigger or a long lock should not be the afternoon. A pipeline that never saw the real shape of the table is a costume rehearsal.
Decision
I want the pipeline to answer three questions: is this artefact fit, are we choosing to release it, and can we undo the parts that are undoable. Fitness is tests plus the boring static checks. Choice is a separate step for anything that talks to students or money. Undo is documented, not implied.
Workers are part of the release. A web node on new code and a worker on old code is a split brain I have already met in mail. The pipeline should make that split hard.
I also want the release to say what it will not do. A pipeline that deploys code and "someone will run the migration" is two processes pretending to be one. Either the migration is in the path, with a lock and a log, or the release is incomplete and should be named that way.
Implementation / Thinking process
Keep install and test deterministic. Pin versions. Cache what you understand. A build that depends on the internet's mood is not CI.
Do not hide deploy in the same job as test without a name. "CI passed" and "production changed" should be different sentences in the log.
Migrations run with a plan: expand first, deploy code that can live with both shapes, contract later. If someone needs a flag-day column rename, that is a product pause, not a clever YAML.
Secrets come from one store the server and the runner agree on. If a human can deploy with a .env they edited by hand, you have a second pipeline.
After release, look at the paths that would embarrass you: login, a payment return, a queued send. That is not a full QA theatre. It is a refusal to walk away from the room.
Feature flags belong in this path when the risk is behavioural, not structural. A flag can hide a screen. It cannot hide a destructive migration. Mixing those two ideas is how teams "roll back" a flag and still live with a column that cannot go back.
Environment parity is a gradient, not a religion. I do not need staging to send real mail. I do need it to run the same job names, the same config keys, and the same migrate command. If the keys differ, you will discover a missing secret in the only place it hurts.
I want the release notes to mention data. "Code deployed" is incomplete if a migrate ran, a flag flipped, or a worker binary changed. Those are different undo stories. If the chat only says shipped, you will attempt the wrong undo. The pipeline can print the list. A human can still press the button. The list is the product.
Failure modes
The skipped check "just this once." There is no once. There is a new default.
A rollback that rolls the code and not the data, or the data and not the workers.
A staging that uses a shared database with production names. You will migrate the wrong place. I will not invent a story. I will say the design invites it.
A pipeline so privileged it can do anything, owned by no one, with a token in a fork pull request. You have automated a breach.
The green check that required skipping a flaky test. You did not buy speed. You bought a new meaning for green. The next person will skip a real one and cite your precedent.
Operational consequences
When delivery is trusted, people ship smaller things. That is the cultural prize. When it is not, people batch "to be safe" and invent the risky deploy they were trying to avoid.
Non-engineers will start asking "is it on production." If you cannot answer without SSH, the pipeline is not yet a product.
You will also discover that failed deploys are a product of their own. A half-applied migrate, a worker that did not restart, a config cache that still has last week's secret name — those are not "the pipeline being flaky." They are states. If the only recovery is "run it again," you will run it again against a world that is already half new. Name the failed states in the runbook before you need them at 18:40.
Lessons
CI/CD is not a badge. It is a way of making change to a live system less theatrical. The artefact, the decision, and the undo are the plot. YAML is scenery.
If the team does not believe the green check, you do not have automation. You have decoration. Belief is the SLO of a pipeline. Without it, people invent a second, unofficial release path, and that path is the one that will hurt you.
What I would do differently today
I would have split test and release on the first production app, not after the first confusing rollback. Combined workflows hide the decision.
I would have treated worker restarts as a first-class step instead of a footnote in a README. The README is not on call.
I would have written the expand/contract rule on the first column rename, not after the first failed undo. A rename that needs downtime is a release type. If the pipeline cannot say "this is a flag day," people will treat it like a Thursday deploy and be surprised when Thursday cannot go backwards.
Closing thought
A safe release is one you can describe: what we proved, what we shipped, what we can still undo, and who we will surprise. If your pipeline cannot say those sentences, it is typing. Typing is useful. It is not the same as being allowed to change the week.