Engineering Journal · ENGINEERING · December 3, 2024 · 16 min read
The Cache That Lied Faster Than the Database
Caching is a second copy of the truth with a shorter memory. If you cannot say when it becomes false, you have not made the system faster. You have made it more confident.
By Golam Sorwar, Tech Lead and Full Stack Engineer in Dublin.
I have reached for Redis the way some people reach for a bigger instance: as a way to stop thinking about a slow question. Put the answer in a key. Give it a TTL. Feel modern. The next request is fast. The request after a write is sometimes still the old answer, and now you have a product discussion about whether the screen is "wrong" or "eventually fine."
I already argued that a slow page is often a foolish query, not an undersized database. This is the sibling argument. Even after the query is honest, caching is not free speed. It is a consistency project you may not have staffed.
A cache that you cannot invalidate in a sentence is a rumour with a stopwatch.
Context
Laravel applications accumulate Redis the way they accumulate helpers. Cache, sessions, queues, rate limits, a lock around a webhook. One instance, several jobs. That consolidation is tidy until the jobs disagree about how important durability is.
The screens that attract cache are the obvious ones: dashboards, navigation counts, catalogue-like lists, permission-heavy menus. In a college admin, that might be an inbox count. In a multi-tenant storefront, a category tree. The user-facing win is real. So is the ticket that says the number is wrong.
I have also seen Redis used as a meeting-avoidance tool. The query was ugly. Nobody wanted to own a read model. A key felt cheaper than a conversation. Six months later the key is in four writers and nobody can flush it without a ceremony. The conversation still happens. It is just worse.
The problem
The apparent problem is latency. The database is doing the same work too often.
The real problem is whether the answer is allowed to be old, and for whom. A public marketing page can be old. A payment balance cannot. A permission bit cannot. A tenant-scoped list that forgot the tenant in the key cannot.
If you cannot say the allowed staleness, you are not choosing a TTL. You are gambling.
There is a second real problem: ownership of the copy. The database has an owner — a migration, a model, a team. A cache key is often owned by whoever was last embarrassed by a slow page. That person leaves. The key remains. The next writer does not know the key exists. You now have an unowned second store. Unowned stores do not invalidate. They haunt.
The tempting solution
Cache::remember with a round number. Five minutes. One hour. Forever, plus a hope that you will forget the key on write. Hope is not invalidation.
Or cache the entire graph of a page because the page is slow. Now every write in the graph is a reason to flush a lake, so you flush nothing, and the lake goes green.
Why that is not enough
Invalidation is the product. Reads are the easy half. Writes happen in more places than the controller you remembered: jobs, admin, imports, another app. If any writer is outside the forget() call, you have a lie with a longer TTL than your attention.
Stampedes happen when a hot key expires and every request rebuilds it. You wanted less load. You created a cliff.
Redis as a second source of truth is how you get split brain. Session in Redis, user in MySQL, a permission cached between them. A change in one place is a ghost in another.
Locks and rate limits deserve more respect than "we already have Redis." A cache that evicts under memory pressure will drop your lock or your limiter if you mixed them with disposable keys. Eviction is not a policy you want for a lock.
There is also the write path people forget: "we will delete the key in the observer." Observers are easy to skip. A raw update, a migration, an artisan command, an import, a second app on the same tables — none of them will fire your observer unless you made them. Invalidation that lives in one happy controller is a wish about how writes will happen.
Options
Do not cache. Fix the query or take the report off the request. Advantage: one truth. Disadvantage: you may still be slow. Often the right first option.
Cache with a TTL only, no write invalidation, on data that may be old. Advantage: simple. Disadvantage: you must be honest in the UI that the number is as of a time.
Cache with explicit invalidation on the writes you own. Advantage: fresher. Disadvantage: every new writer is a bug. This needs ownership, not a helper.
A snapshot table or a projected read model updated by the same transaction or a job. Advantage: you can index for the page. Disadvantage: you are doing the hard work people pretend Redis avoids.
Separate Redis roles. One instance for disposable cache, another for queues and locks, or at least separate prefixes and eviction policies. Advantage: a cache flush does not become an outage. Disadvantage: more things to run. I would rather have two small honest instances than one clever soup.
Trade-offs
You trade a little staleness for a lot of load, or you trade a little load for a lot of invalidation code. There is no third thing, only people who have not met the second bill yet.
You trade operational simplicity. Redis is now in your incident story. "Is Redis up" becomes as important as "is MySQL up," except Redis failures look like random wrongness rather than a clean down page.
You should not trade correctness on money, tenancy, or auth for a faster nav badge. Those facts can be slow. They cannot be approximately someone else's.
Decision
I cache when I can say, in one sentence, what is stored, who may see it, and which writes make it false. If I cannot, I do not cache. I fix the question or I accept the time.
I keep queues and locks out of the same eviction story as page fragments. That is not purity. That is not wanting a dashboard cache to evict a payment lock.
I also refuse to cache as a way of hiding N+1. If the page issues a hundred queries, a cache will hide the shame until the key expires at the worst moment. Fix the shape first. I have already made that argument about bigger databases. It applies here with a shorter TTL.
Implementation / Thinking process
Keys include tenant, role, and the meaning of the number, not just a table name. A key called users is a confession.
Prefer small payloads. A cached HTML page is a hostage situation when a footer changes. A cached count is a fact.
Invalidate next to the write, in the same module that owns the fact, not in a random observer someone will delete. If two modules write the fact, the cache is already telling you the ownership is wrong.
For stampedes, a lock around rebuild or a slightly staggered TTL is dull and effective. Clever probabilistic expiry is optional.
Measure hits if you want, but measure wrongness more. Support tickets about stale numbers are the real hit rate.
Be careful with "warm the cache on deploy." Warming can stampede just as well as expiry, and it can warm the wrong tenant if your seeder is naive. I would rather a slow first request than a clever warm that writes one customer's page into a global key.
Sessions deserve a separate sentence. Putting sessions in Redis is not the same as caching a dashboard. A session flush logs people out. A dashboard flush makes a number blink. If those share a FLUSH command in a runbook, the runbook is a weapon.
I also want a written owner per family of keys. Dashboard counts are one family. Permission bits are another. They do not share a TTL, a flush, or a "just cache it" instinct. If a family cannot name who invalidates it, it does not get a key. That rule sounds harsh until you have flushed the wrong family to fix a badge.
Failure modes
The key that forgot the tenant. I have mentioned this in tenancy. It belongs here too. Fast leakage is worse than slow leakage.
The flush-all that a developer runs in production because a key was stubborn. You have just logged everyone out or emptied a queue, depending on how sociable your Redis is.
A negative cache — "not found" — that outlives the create. New records appear to not exist. This feels haunted.
Caching authorised data under a key that is only the object id. The next user gets a faster view of someone else's inbox.
Caching a computed permission and then changing a role in the database. The person is no longer staff. The key disagrees for the rest of the TTL. That is not eventual consistency. That is a delayed "yes" on a door you already closed.
Operational consequences
Once a cache exists, people will put more in it. The first key is a gateway drug. You need a written list of what is allowed to live there. If the list is "whatever is slow," you will cache a permission check and meet it in an incident.
Incidents involving caches are social. The database is right. The screen is wrong. Engineering looks incompetent while being, in a narrow sense, fast.
The other consequence is diagnostic fog. When a number is wrong, the first question is now "is it stale or is it false." That question did not exist when there was one store. You will spend meetings on it. If you cannot answer it from a header, a timestamp on the payload, or a "as of" in the UI, you will answer it with a flush, and the flush will become the culture.
Lessons
A cache is a second database you are pretending not to operate. If you would not accept that database being wrong, do not put the fact in Redis.
Speed is not the same as freshness. Choose which one the screen is selling.
What I would do differently today
I would have separated cache from queue earlier, even when the bill looked silly. Mixed Redis is a rite of passage I do not need to repeat.
I would have required the invalidation sentence in the pull request, next to the remember() call. No sentence, no cache. TTL-only is allowed if the sentence is "this may be five minutes late and that is fine."
I would have put an "as of" on any cached number a human might argue with. A badge without a time is a dare. A badge with a time is a fact people can live with. The extra words in the UI are cheaper than the ticket that says the system is lying.
Closing thought
Redis will happily remember a lie. That is its talent. Use it when the lie is bounded and named. If the only thing you know is that the page was slow, you are not ready to add a second truth. You are ready to ask a better question of the first one.