Engineering Journal · INFRASTRUCTURE · January 27, 2026 · 16 min read
The Dashboard Was Green and We Still Did Not Know
Uptime is a narrow kind of knowledge. Observability is being able to ask the next question — including the operational one — when the boxes look fine.
By Golam Sorwar, Tech Lead and Full Stack Engineer in Dublin.
I have sat in front of a green board and a red conversation. CPU was calm. Disk was calm. The application was "up." A person in operations was not asking about uptime. They were asking why last night's letters had not gone, or why a payment was paid on one screen and not on another, or why a queue of human work had doubled.
We had monitoring. We did not have understanding.
I already wrote about reconstructing a single incident from incomplete evidence. This is the layer above that. What do you choose to see before the week goes strange, so the reconstruction is possible and the alert is about work, not about a machine being slightly warm.
Context
The stack is ordinary: application logs, a host or container view, maybe a few Laravel pulse-style numbers, queue depths if someone bookmarked Horizon. Vendors have their own dashboards. The business has spreadsheets.
None of these is wrong. The failure is when we treat host green as a proxy for "the organisation can do Tuesday."
I also care about who can read the board. If only one engineer can interpret the charts, you have not bought observability. You have bought a private language. The useful test is whether a deputy can answer "is the work moving" without calling you. That is the same holiday problem as folklore, wearing a prettier UI.
The problem
The apparent problem is lack of tooling. We should buy APM.
The real problem is lack of questions. If you do not know what you would ask, you will buy charts that answer "how busy is PHP." That is a systems question. Operational software also needs: how many intents are stuck, how old is the oldest mail, how many payments are captured and unallocated, how many jobs died since noon, did error rate change after the deploy.
If those questions live only in a person's head, you have a green dashboard and a human SLI.
The problem is also attention. People will manage what the board shows. If the board shows hosts, they will manage hosts. If the board shows stuck letters and unallocated payments, they will manage those. Observability is therefore a choice about what the organisation is allowed to ignore. A green tile is a permission to look away.
The tempting solution
Turn on everything. Log every request. Alert on every 5xx. Watch CPU. Feel covered.
Or alert on nothing until it hurts, then alert on that forever. The next month you have a noisy room and a muted channel.
Why that is not enough
Request logs without business identifiers are a firehose. You will not drink from them when you are tired.
Alerts that are not actionable become decoration. If the response is "we knew, it does that," you have trained the team to ignore the next alert, which might be the one.
Traces are wonderful for a slow endpoint and silent about a job that never ran.
Vendor dashboards are not your product's health. They are a neighbour's opinion. I have written about neighbours. Observability includes not believing them alone.
Business events without technical context are the other half of the lie. "Letters sent: 400" does not say whether that is the cohort you meant, whether the oldest is six hours late, or whether the worker is chewing a retry storm. A count without age is a press release. Operations need the lag.
Options
Host metrics only. Advantage: cheap, familiar. Disadvantage: you will miss the quiet functional failures — the empty send, the stuck sync.
Full platform, every service traced. Advantage: you can follow a request. Disadvantage: cost and noise if you have not named the golden paths.
A short list of user-meaningful signals, plus enough technical detail to debug them, plus alerts that imply a next action. Advantage: the board talks about work. Disadvantage: you have to maintain the list as the product changes. That is the job.
I want the third. Buy or build tools in service of it, not the other way around.
There is a fourth option I see too often: copy a vendor's "golden signals" poster and stop thinking. Latency, traffic, errors, saturation are a good start for a request-shaped service. They are a poor finish for a system whose pain is a job that never ran, a webhook that never arrived, or a human queue that doubled while HTTP stayed pretty. Use the poster as a floor, then add the signals that match how the organisation fails.
A weekly human review of stuck work and no new tools. Advantage: you will learn the questions. Disadvantage: it does not page you at 2am. Use it to decide what to instrument. Do not use it as the only nervous system once you know the sentence you care about.
Trade-offs
You trade volume of data for quality of questions. Less pretty, more useful.
You trade some engineering time to emit events — payment.allocated, mail.accepted, tenant.context.missing — that are not features. They are how features stay honest.
You should not trade away privacy. Observability that stores message bodies "just in case" is a second product with a worse privacy policy. A trail can name the intent without quoting the letter.
Decision
I instrument the paths whose failure is an organisational sentence, not a graph. Login, money, the outbound voice of the institution, the jobs that feed those. I want age and count of stuck work, not only throughput.
I alert when a human should do something within a bounded time. I do not alert because a number is interesting. Throughput without age is how a busy worker hides a dead one.
I also want a distinction between "the process is up" and "the work is moving." Those can diverge for a whole afternoon. If the only page you have is the first sentence, you will reassure people while the second sentence is already false.
Implementation / Thinking process
Start from the last painful questions, even if you will not invent the stories around them. What did we wish we knew. Put a number on that. If you cannot name the number, you are not ready for a chart. You are ready for a sentence. Charts come after that sentence already has a name.
Put the same identifiers everywhere: tenant, user, intent id, provider id. Correlation is the difference between a log and a trail.
Separate saturation (the box is full) from correctness (the work did not finish). Both can be red. They are different on-call.
For queues, depth without age will lie. A deep queue that is moving is a busy day. A shallow queue whose oldest job is six hours old is a stuck day.
Review alerts the way you review code. If nobody can say the action, delete or rewrite. An alert is a user interface for a tired person.
I also want a weekly look at the questions we still cannot answer. Observability work that only happens during incidents is just debugging with a nicer bill. The cheap time is the quiet week when you notice that mail age is invisible, or that payment stuckness is a SQL query someone runs by hand. Put that query on a board. Give it a threshold. Then you can spend the incident on the cause instead of on inventing the question.
Do not forget the human layer. A comment that says "retried, still pending" is an event. If those comments never leave chat, your system of record for operations is Slack. I would rather a small status on the intent than a long thread you cannot query.
I pick a handful of golden paths and give each a sentence: login works, money can be explained, the institution can still speak, stuck work has an age. Then I attach the smallest signal that would falsify the sentence. That is the board. Everything else is optional decoration until those sentences are cheap to ask.
Failure modes
The vanity dashboard. Twelve charts, no owner, opened for visitors.
The alert that fires on deploy every time. You will ship less, or mute it, and you will deserve both.
The metric that used a cache and then you cached the metric. You are watching a memory of health.
The log level that was debug in production for a week and cost you a disk, or a bill, or a leak.
The business metric that was implemented as a cached page fragment. You are now observing the cache. When the fragment is stale, the board is calm and the office is not. That is the green-tile problem wearing a prettier name.
Operational consequences
When the board can answer "is the work moving," conversations with non-engineers get shorter. You can say the mail age is fine and the payment intent is not. That is a better sentence than "the server is fine."
When it cannot, engineering becomes a priesthood that interprets green. Priesthoods do not scale, and they do not take holidays well.
There is a quieter consequence: you will overreact to the things you can see and underreact to the things you cannot. A CPU spike gets a meeting. A growing pile of unallocated payments gets a shrug until finance writes. That is not because people are careless. It is because attention follows the dashboard. Build the dashboard you want the organisation to care about.
On-call changes when alerts imply actions. A page that says "mail age > 30 minutes, check the failed provider and the worker" is a job. A page that says "error rate +12%" is a riddle. Riddles get muted. Jobs get done. If you cannot write the action, you are not ready to page.
Lessons
Monitoring is having numbers. Observability is being able to ask the next question, including "can the organisation still do the thing we sold them."
Green is a colour. Understanding is a sentence. If nobody on the team can say the sentence without opening five tabs, you do not have a dashboard. You have a collage. Collages do not page. Sentences do, if you give them a threshold and an owner.
What I would do differently today
I would have defined stuck-work views before I added another host graph. I liked graphs. Operations liked unfinished letters.
I would have given each alert an owner in the same sentence as the threshold. Unowned alerts are how channels die.
I would have put age next to every queue depth on the first board, not after we learned that a shallow stuck queue is worse than a deep moving one. Depth is a vanity metric for work that can stall.
Closing thought
If the only thing your dashboard can say is that the process is running, you have a heartbeat, not a nervous system. A heartbeat is necessary. It is a poor narrator for a week of work. Instrument the work. Alert on the action. Leave the green tiles for the things that are actually up, and stop asking them to speak for the week.