The 15-min limit is a signal the problem needs rethinking — not the infrastructure scaling up.
"This job takes X hours → we need a long-running container / VM / ACI to hold it."
"Why does it take X hours? Move to write-time, fan-out, or checkpoint through a queue."
Whenever he bought vanilla ice cream, his car wouldn’t start when he got back to it.
Chocolate? Fine.
Strawberry? Fine.
Vanilla? Dead car. Every time.
Most companies would’ve told him to stop drinking on the job. Pontiac sent an engineer instead.
Turns out vanilla was the store’s most popular flavour, so it sat in its own freezer right at the front. Grab it and go: two minutes, tops.
Every other flavour was further inside, at a different counter. Longer walk, longer wait.
That extra time let the hot engine cool down. On the quick vanilla trip there wasn’t enough of it. The fuel in the lines was still hot, hot enough to turn into vapour and block the flow to the engine. This is called vapour lock.
The problem was never vanilla. It was the time between stopping the car and restarting it.
Vanilla didn’t cause anything. It just happened to be standing at the front of the shop.
(Real story or engineering legend, nobody’s fully sure. Doesn’t change the lesson.)
When something breaks, we tend to mix three very different things into one.
What did we actually observe?
“When I buy vanilla, the car doesn’t start.”
That’s a fact. We can reproduce it, measure it, test it. Unlike the other two, it doesn’t change depending on who is in the room.
How does it make us feel?
“This is scary. I just bought this car and now something’s wrong with it.”
Completely real, and completely useless as evidence. The worry is what makes people rush; it can’t tell you anything about the car.
What do we think is causing it?
“Something about vanilla is messing with the engine.”
A guess dressed up as an explanation. It might turn out to be right. It might turn out to be the thing you quietly stop mentioning.
The trouble starts the moment we treat the guess as a fact.
A production system goes down.
“It’s the database.”
“No, it’s the network.”
“It started right after the deploy — so it must be the deploy.”
Five minutes later, the team is arguing about solutions to a problem nobody has actually confirmed.
The only real fact on the table is usually something like:
“Requests started timing out around 10:35.”
And that is all we actually know. Database, network, deploy, traffic spike, cache, race condition: every one of those is still a guess, just delivered with confidence.
“We don’t know yet” is a perfectly good engineering answer. Pretending otherwise is how you spend four hours fixing something that was never broken.
This is the whole point of the vanilla story, and the hardest habit to build, because the explanation is always more interesting than the observation.
Someone says:
“It only happens with customer X.”
Don’t rush to check customer X’s account like it’s cursed. Ask instead:
What’s actually different when we process customer X?
Bigger data? A different code path? A specific time of day? A setting only they use?
Customer X might simply be your vanilla ice cream. The pattern is real; the explanation might not be.
The engineer Pontiac sent didn’t investigate vanilla. He went shopping, three times, with a stopwatch. What he was really doing was looking for what co-varies with the failure, and the answer turned out to be elapsed time, which nobody had thought to write down because nobody thought it mattered.
The move is to take the thing that supposedly explains the failure and list everything that travels with it:
Now you have six testable statements instead of one unfalsifiable one. Any of them can be checked in an afternoon. “It’s customer X” can’t be checked at all, which is exactly why it survives so long in a chat thread.
I ran into this on a financial processing pipeline for short-term rentals. Transactions arrived on an SQS queue, and a Lambda function sat on that queue and drained it.
One client’s work took ages. Not failing — just crawling, for hours, while every other client went through in minutes.
And that client already had a reputation. They’d been a source of strange requests for months. Unrelated things broke on their account. So by the time this landed, “it’s them again” wasn’t a theory anyone had to argue for. It felt earned. It genuinely felt like a curse, and I believed it for longer than I’d like to admit.
That reputation was our vanilla ice cream. The pattern was completely real — it was always them — and the explanation was completely wrong.
What was actually different about them had nothing to do with who they were. It was how they submitted work. Every other client accumulated transactions per day, running totals rolling forward. This one submitted an entire year of them in one go, which hit the queue as a single enormous burst of messages.
So the load we had tested, repeatedly and carefully and many times over, covered every shape except that one. Small volumes: fine. Large volumes spread across days: fine. A year of financial transactions arriving all at once: never tested, because nobody had thought of it as a case that existed.
And that Lambda’s concurrency was capped at one. Not a decision anybody had made: it was whatever the function had been deployed with on day one, and nobody had revisited it since. One invocation at a time, every message waiting for the one in front of it.
And this is the part worth being precise about: nothing was slow. Every individual message was processed quickly, nowhere near any limit. There were simply an enormous number of them, and they went through the queue in single file. At normal volumes that was invisible, because one at a time was plenty and we never came close to the ceiling. With a year of transactions stacked up behind it, the serial approach was the problem.
When I finally sat down and worked through what actually co-varied with the slowness, it wasn’t the client. It was how many messages arrived at once, against how many could be worked on at once. One of those numbers was an order of magnitude past anything we had tested. The other was one.
I raised the concurrency from one to ten, so ten of them ran in parallel instead of one after another. That was the whole remedy, and because Lambda only runs what it’s given, normal operation cost us almost nothing: the extra capacity existed only when the queue was deep enough to need it.
Months of “that client is cursed,” and the answer was a concurrency setting nobody had ever looked at.
I’d like to report that the nickname retired along with the setting. It did not.
What’s uncomfortable in hindsight isn’t the mistake itself, it’s how reasonable we were being while we made it. Every previous failure on that account was real evidence — for the wrong conclusion.
The test that would have settled it in an hour: take a different client’s data, reshape it into a year in a single run, and push it through. If that crawls too, the client is innocent and the volume is the culprit. Nobody ran it, because nobody thought there was anything to test.
Converting a theory into a check is the part that separates debugging from arguing. It usually means finding the one measurement that can only come out one way.
“It must be the deploy — the timeouts started right after it.”
The deploy landed at 10:31. Timeouts began at 10:35. So did the hourly reconciliation job. Both are “right after.” Roll back the deploy and the timeouts should stop immediately; pause the job and they should stop at the top of the next hour. Whichever moves is your answer, and finding out takes about ten minutes. Arguing about which one feels more likely can comfortably fill an afternoon.
“It’s the database.”
Is the database slow, or is it starved of connections because something upstream stopped returning them to the pool? Those look identical from the application and want opposite fixes. Measure at the database, not at the thing complaining about the database.
“It only happens in production.”
Production is not a cause, it’s a bundle of differences: data volume, concurrency, config, network path, TLS termination, someone else’s traffic. Name them, then rule them out one at a time.
Which brings me to the phrase every team has heard, usually from the same person.
We had a developer who said “but it works on my computer” every single time a bug turned up in production. It became a running joke. Someone would open the incident channel and half the team was already typing it before he could.
Here’s the thing, though: he was right. Every time.
It did work on his computer. That isn’t a defence, it’s a measurement — and a genuinely useful one. It tells you the code is capable of working, so whatever is broken lives in the difference between his machine and production. It’s the most valuable sentence anyone can offer in the first five minutes of an incident, and we turned it into a punchline.
“It works on my computer” is exactly “chocolate is fine.” And chocolate was fine. That was the clue. Nobody laughed at the Pontiac owner for mentioning it; an engineer wrote it down and went shopping.
What we should have done, every time he said it, was ask the follow-up: then what’s different about yours? Java version, a stale local config, a database with fifty rows instead of fifty million, no concurrent traffic, a mock standing in for something real, a timezone set to his own.
Most of those differences are accidents. Some are designed in, which is worse.
The joke got a laugh. The question would have got us an answer.
The first wrong theory is cheap: you lose an afternoon. The habit is what gets expensive, because wrong explanations don’t stay in the incident channel. They get written down.
They become the runbook step everyone follows for the next two years. They become the alert that fires on the symptom you misdiagnosed. They become the retry someone added around the wrong call, the cache in front of a database that was never the problem, the nightly restart nobody dares remove because it “fixed” something in 2023.
Every one of those is a defence built against the vanilla.
And because the real cause was never found, it comes back. Different symptom, different quarter, different on-call engineer — who now has a runbook confidently pointing in the wrong direction, and a system carrying three years of scar tissue from the last four times this happened.
Nobody on the team experiences this as failure. It just feels like working on a hard system that breaks a lot.
Separate, every time:
FACT — What do we know happened?
FEELING — What worry does it create?
ASSUMPTION — What do we think caused it?
Then do the one thing most people skip: test the assumption.
Don’t argue about whether it’s the database, the network, the deploy, or the vendor. Build a small test that can actually prove or kill the theory.
That’s debugging. Everything before it is opinion with confidence attached.
They can turn:
“I think it’s the database.”
into:
“Let’s test if the database is actually responsible.”
And then — the hard part — accept it when they’re wrong. The goal was never to be right; it was to find out what’s actually happening.
This is really what troubleshooting culture is about.
Not finding someone to blame. Not being the person with the fastest answer. It’s about building a habit, as a team, of separating facts from guesses and testing before deciding. Teams that do this learn faster. Teams that don’t will keep repeating the same guesses year after year, and calling it experience.
Whether or not the Pontiac story is 100% true, it points to something that is: the pattern was real, the explanation was wrong. Worth remembering next time your production system does something that makes no sense.
Don’t blame the vanilla. Find out what’s actually different about it.
]]>I watched this happen firsthand. Over a couple of years, our team’s culture shifted from “let’s find the best way to solve this” to “let’s close the ticket.” The tool didn’t do this on its own, but the way we used it did. The damage wasn’t abstract: good people left. The people who stayed were, on average, less capable at the actual craft of engineering than the ones who walked away.
The uncomfortable part is that nobody intended this outcome. We thought we were improving planning, transparency, and accountability. What we were really doing was teaching engineers that visible activity mattered more than thoughtful engineering.
Story pointing is supposed to help teams plan. In practice, on my team, it became a performance metric in disguise.
Every task was sized, every sprint produced a velocity number, and that number was watched by people who didn’t write code but cared deeply about whether it went up or down.
Once that happens, the incentive structure is set.
Nobody explicitly said, “Stop thinking and start closing tickets.” But the system rewarded exactly that behaviour.
Only a closed ticket closes a ticket.
As a result, the smart, curious engineers—the ones who wanted to understand why something was broken instead of merely hiding the symptom—started to look like underperformers. Their velocity appeared lower. Their story-point totals accumulated more slowly.
It didn’t matter that their fixes were more durable, or that their code required less rework later. The dashboard had no column for “prevented next quarter’s outage.”
There used to be a meaningful distinction between working hard and working smart.
Jira flattened that distinction.
Story points don’t measure judgment; they measure throughput. And throughput is surprisingly easy to game if you’re willing to slice work into small enough pieces.
That’s exactly what happened.
A subset of the team—often the more average technical performers—figured out that the game was no longer “solve problems” but “generate tickets.”
A straightforward feature would be split into 30, 40, sometimes more than 100 tiny tasks, many barely worth the overhead of creating them.
Every one of those tickets could be closed. Every closed ticket became a visible unit of “hard work.” The burndown chart looked fantastic.
The software itself did not become significantly better, faster, or easier to maintain.
Meanwhile, an engineer who spent two days understanding a nasty concurrency bug instead of applying a five-minute band-aid appeared, on paper, to have done less work. They had a single ticket sitting in In Progress for days.
To anyone glancing at the board, that looked like underperformance.
In reality, it was often the opposite.
This was the hardest part to watch.
The engineers who cared about doing the work well—the ones who wanted to research an approach, propose a better architecture, or simply have enough uninterrupted time to think—gradually became exhausted.
Some received mediocre performance reviews because their point totals didn’t compare favourably with the ticket farmers. Others simply grew tired of a system that punished depth and rewarded visible busywork.
Eventually, many of them left for teams that measured outcomes instead of ticket counts.
Who stayed?
Mostly the people who had adapted to the system rather than resisted it. Not bad people—just people who had correctly learned that the safest path to looking successful was to generate volume:
Over time, the team’s average technical judgment declined even as its reported velocity increased.
We became extremely efficient at closing tickets and noticeably less effective at deciding which tickets were worth creating in the first place.
If ticket-gaming is a subtle problem on an in-house team, it becomes a structural one the moment you bring in an outside vendor.
I’ve seen this pattern repeatedly on outsourced projects, regardless of which country the vendor was based in.
The client-side stakeholders—usually directors or VPs several layers removed from the code—don’t read diffs and don’t sit in on debugging sessions. What they see is the Jira dashboard:
That dashboard becomes the proof of value for a contract that may be costing hundreds of thousands of dollars per year.
Vendors optimising for contract renewal rather than codebase health learn very quickly that the dashboard is the product.
So the same behaviour appears again, only amplified:
The metrics look impressive. The architecture quietly deteriorates underneath.
Technical debt accumulates because technical debt is not what the dashboard measures.
The result is a slow-motion divergence between the story the metrics tell and the story the codebase tells.
Nobody can point to the exact moment things went wrong because the reporting system never showed a decline.
It couldn’t.
It was never measuring the thing that mattered.
This is why so many outsourced software projects either fail outright or quietly bleed value for years before anyone notices: ticket throughput was never a good proxy for a maintainable, working product.
The vendor may not even be acting in bad faith. They are often responding rationally to a contract that rewards visible activity rather than engineering judgment.
Rational responses to bad incentives still produce bad outcomes.
To be fair, Jira didn’t force anyone to behave this way.
A healthy engineering culture with thoughtful leadership can absolutely use Jira while still protecting space for research, deep work, and technical judgment.
The failure was organisational, not purely technological.
But I also don’t think the tool is entirely neutral.
Jira’s design vocabulary—story points, velocity charts, burndown graphs, sprint reports—naturally encourages people to treat ticket throughput as a proxy for value delivered.
That proxy is wrong often enough, and quietly enough, that many organisations never realise they have optimised for the wrong thing until their strongest engineers are gone and the remaining team is composed of people who are excellent at Jira and only average at engineering.
If I were leading that team again, I wouldn’t ban Jira.
I would ban the assumption that Jira metrics tell you who is contributing the most.
Planning tools are useful for coordinating work. They are dangerous when they become surrogate performance-review systems.
I would explicitly create protected space for work that rarely looks impressive on a board:
I would also treat unusually high ticket throughput as a signal to investigate, not automatically as evidence of exceptional performance.
Sometimes the most productive engineer on the team is the one with fewer tickets closed, because they are solving the problems that prevent everyone else from drowning in tickets later.
The biggest cost of optimising for Jira metrics is not bad charts, inflated velocity, or bloated backlogs.
It is the gradual loss of the people who care about understanding systems deeply.
Those engineers are often the ones who:
They are also the people most likely to become frustrated by environments that reward visible busyness over thoughtful engineering.
If your dashboards celebrate ticket volume more than customer outcomes, system reliability, or long-term maintainability, you may already be training your team to optimise for the wrong thing.
And by the time the metrics finally stop looking good, the people who could have warned you about it may already be working somewhere else.
You’ll still have plenty of tickets getting closed.
What you may no longer have is a team that knows which tickets are worth solving, which problems are worth understanding, and which shortcuts will become next year’s disaster.
That is a much harder thing to measure than velocity.
And a much more expensive thing to lose.
]]>Then you open the repository and find docker-compose.yml. Postgres. RabbitMQ. Redis. A .env file somebody maintains by hand. A folder called mocks that has grown a maintainer.
Both of those things are true at the same time, and nobody experiences this as a contradiction. The diagram describes what we deploy. The Compose file describes what we develop. The gap between them is where the interesting failures live, and we only ever meet them in production, because production is the first place the two descriptions are forced to agree.
This is the pattern I keep running into, and I’ve stopped thinking of it as a tooling problem. It’s on-premises development wearing a cloud badge.
The uncomfortable part — the part that took me a while to accept — is that this is not what inexperience looks like. Junior developers use whatever the onboarding doc tells them to use. The local-first habit belongs to the senior people, because it’s built out of a decade of genuinely hard-won instinct: run it yourself, keep it close, don’t depend on infrastructure you can’t restart. Every one of those instincts was correct in a data centre. Several of them are now the reason the thing we tested isn’t the thing we built.
Good judgement, applied to the wrong decade, is still going to hurt you.
What follows is the list of habits, and what I think we should do instead. The examples lean AWS because that’s where I’ve spent most of my time, but none of the reasoning is AWS-specific — swap in Azure or GCP service names and the argument is unchanged.
“I do my dev locally — it’s faster.”
If your company pays for an AWS Dev account, that account is your development environment. Not a deployment target you visit at the end of the sprint. The environment.
Put it in the same region as your users and your corporate network. Developing in Melbourne against ap-southeast-4, the path is:
Laptop → corporate network → AWS ap-southeast-4 (Melbourne) → Dev services
The latency on that path is low enough that the cloud behaves much more like an internal development network than the phrase “remote server” leads people to expect. Most of the intuition people have about remote development was formed on a VPN to a data centre on another continent, and it has not been updated since. Sydney’s ap-southeast-2 is close enough to be the fallback if a service you need isn’t in Melbourne yet; on Azure the equivalent is Australia Southeast.
Developing this way buys the single biggest advantage cloud development has to offer: you are building against the platform you are deploying to.
If production uses DynamoDB, develop against DynamoDB. If production uses SQS, develop against SQS. If it’s Cosmos DB and Service Bus, use Cosmos DB and Service Bus. What you should not do is quietly substitute:
None of that is faster development. It’s a miniature on-premises environment on your laptop, plus the hope that it behaves like the cloud.
Local still has its place — code with no cloud dependencies, offline work, quick experiments, and the cases where an official emulator genuinely does the job (DynamoDB Local is the honourable example: same API, same vendor, documented differences). The problem is local infrastructure as the default, chosen for no better reason than that the real infrastructure lives somewhere else.
Local code, real cloud services, a Dev account in the right region.
Your laptop is the development workstation. The cloud Dev account is the development environment. Conflating those two is where most of this begins.
“Let’s use RabbitMQ locally — everyone knows it.”
Production uses Amazon SQS. The developer runs RabbitMQ. Those are two different systems, and no amount of interface abstraction changes that.
RabbitMQ is not a bad message broker. Using it as a stand-in for the broker you’re actually shipping is the bad part. Delivery guarantees differ. Ordering semantics differ — a standard SQS queue does not promise order at all, and a FIFO queue is a different product with its own throughput limits. Dead-lettering differs. Visibility timeouts have no RabbitMQ equivalent. Retry and back-off behaviour differs. Throttling under load differs.
Every one of those differences is invisible until the system is under real load, which is to say: until customers are the ones discovering it. Your abstraction layer will faithfully hide the differences right up to the moment they matter, and then it will hide the cause too.
Do this instead: develop against the messaging service you’re shipping. And if RabbitMQ really is the right choice, choose it for production and run it properly — the anti-practice is the mismatch, not the technology.
“Cloud functions are too complex — I’ll use Docker.”
Serverless runtimes have real local tooling, built by the platform vendors, specifically so you don’t have to do this. Lambda has the SAM CLI — sam local invoke runs your handler in the same runtime image Lambda uses. Azure Functions has Core Tools. They exist and they work.
Replacing the runtime with a hand-rolled Docker architecture because containers feel more familiar doesn’t remove complexity. It relocates it — out of a runtime that somebody else maintains, documents, patches and supports, and into a Dockerfile that you now own forever, alone, undocumented.
Do this instead: use the official local tooling, and integrate against the real services in your Dev environment.
“We’ll deploy this temporarily. We’ll fix it properly later.”
Watch the life of a temporary deployment:
temporary → feature → dependency → production → legacy
Not one step in that chain requires a decision. It happens on its own, by default, while everybody is busy with something else. The only conscious decision anyone ever makes is the first one — and it gets made with the word “temporarily” attached, precisely to lower the apparent stakes of making it.
By the time something else depends on it, “we’ll fix it properly later” has become a migration project with a budget and an owner who didn’t choose it.
Do this instead: before you deploy the temporary thing, ask the only question that matters.
“Would I be happy if this were still running in 18 months?”
If the answer is no, fix it now, while it’s cheap and nothing depends on it.
“I don’t have cloud access — that’s why I started locally.”
This one deserves genuine sympathy, because it usually isn’t the developer’s fault. Someone needed to ship, the access request was sitting in a queue, and they did the reasonable thing with what they had.
But look at what got substituted for what. The problem was:
“I can’t access AWS.”
And the thing that got built was:
“Let’s not use AWS.”
An access ticket became an architecture. The ticket would have taken a week. The architecture will outlive several people’s employment, and nobody will remember it started as a permissions delay.
Do this instead: treat missing cloud access as the delivery blocker it is, and escalate it as one. Access problems get fixed in days when somebody insists. Architectures built to route around them get fixed in years, if ever.
“I don’t understand the cloud service — the old tech feels safer. I already know Postgres.”
I believe you. That still isn’t an architecture decision.
The tell is that the reasoning describes the state of your knowledge rather than any property of the system you’re building. “Postgres suits this access pattern” is an architecture argument. “I already know Postgres” is a fact about you, dressed up as a fact about the system.
Comfort decisions deserve to be taken seriously — unfamiliar technology carries real risk, and pretending otherwise is how teams end up operating something nobody understands. But the honest version of the argument is “I don’t know this yet,” and the honest response to that is to go and learn it. Ask a colleague. Read the docs. Find a working example. Pair with someone who’s done it. Use AI to compress the learning curve, which is one of the things it is genuinely excellent at.
Do this instead: spend two days learning the right technology rather than two years maintaining the wrong one. The exchange rate is not close.
“If it works, there’s no need to fix it.”
It is also entirely compatible with:
Sometimes the right answer really is to leave it alone. Stability has enormous value, and rewriting working systems for aesthetic reasons is its own anti-practice. The problem isn’t leaving things alone — it’s leaving them alone by default, without ever having looked.
Do this instead: periodically ask:
“Would we choose this architecture today?”
“No, but it’s fine” is a perfectly good answer. “No, and I hadn’t thought about it” is the one to worry about.
“I asked Copilot and it generated the whole thing.”
You asked for this:
“Make it run locally.”
You got Postgres, RabbitMQ, Redis and a Docker Compose file.
Of course you did. You asked for a local system and it built you a local system, competently, in about four seconds. The model is not confused about cloud architecture. The prompt encoded the wrong architecture and the model implemented it faithfully, which is the one thing we all claim to want from these tools.
Try telling it what you’re actually building:
“Build a cloud-native solution using Lambda, DynamoDB and SQS, with local development through the SAM CLI and integration against our Dev account.”
Same tool, same four seconds, a fundamentally different result.
Think first, prompt second. And since this is now most of how code gets written:
You generated it → you understand it → you deploy it → you own it.
If you can’t explain what it does and deploy it yourself, it is not ready to hand to somebody else to operate at 3am.
“Let’s do a monorepo — it’s easier for AI agents.”
This one is new, and I’m flagging it precisely because it sounds modern enough to pass unchallenged in a design review. Nobody wants to be the person who objected to the AI-friendly option.
It still isn’t an architecture reason. Repository structure should follow how software is built, owned and released. If services are developed and deployed independently, a polyrepo with properly versioned shared packages is often simpler and safer. If components genuinely belong together and release together, a monorepo may be exactly right — plenty of them are.
Both are defensible. “The AI likes it” is not one of the defences. We spent twenty years learning to structure repositories around teams and release boundaries; it would be a strange time to start structuring them around a tool’s context window.
Do this instead: choose repository structure for the architecture and the teams who own it. Tooling ergonomics — human or artificial — is a tiebreaker at most.
.gitignore Is Not a Security Architecture“Secrets live in .env — it’s in .gitignore.”
.gitignore is a list of files Git ignores. That is a much smaller promise than the one being relied on.
It does nothing about the copy on your disk, the copy in the backup, the copy pasted into Slack while you were helping someone debug, or the afternoon someone renames the file and the pattern silently stops matching.
The platforms solved this. AWS has IAM roles, Secrets Manager and Parameter Store. Azure has Managed Identity and Key Vault. A role your code assumes is strictly better than a secret, because there is no secret to leak — and long-lived access keys in a .env file are the AWS-specific version of this mistake.
Do this instead: no credentials in source, config files or Compose files — including the ones that are “only for development.” Development credentials reach production surprisingly often, usually by being reused.
“We’ll just click it up in the console.”
ClickOps genuinely is faster. It is faster right up until the first time you need to reproduce the environment, stand up a second one, recover from a disaster, explain why staging behaves differently from production, or work out who changed that setting and when.
At that point the configuration exists in exactly one place: the running system. The only way to read it is to click through the same forty console screens that created it, and the only record of intent is whatever people remember. CloudTrail will tell you who changed it; it will not tell you what they meant.
CloudFormation, CDK, Terraform — pick whichever suits the team.
Do this instead: infrastructure should be reproducible from code. The console is an excellent tool for operating and inspecting infrastructure. IaC is the source of truth.
“Testing in prod is fine — we move fast.”
It isn’t fine, and that phrase is carrying an enormous amount of weight: no staging environment, no test coverage, and no rollback plan, compressed into three words that sound like a culture rather than an omission. Every deployment becomes a bet placed with customer data, and the customer finds out before you do.
Canary releases, progressive rollouts and feature flags are not testing in prod. They are controlled, monitored exposure of already-tested code, with a rollback path established before the first customer sees anything. The entire premise is that you already believe the code works and are limiting the cost of being wrong.
Conflating the two is how a legitimate release strategy becomes cover for skipping the testing — you get the vocabulary of sophisticated delivery practice and the risk profile of deploying on a Friday afternoon and going home.
Do this instead: test before it ships. Use staged rollouts to limit blast radius at release, not as a substitute for having tested.
Almost every habit above survives for the same reason: nobody said it out loud in a room with other engineers in it. Local Postgres doesn’t get proposed and defended. It gets committed on a Tuesday.
None of these are hard questions. That is rather the point.
The obvious cost is the incident — the release that behaved differently in production because production was the first environment where the real services were involved. That one is at least visible. Somebody writes it up.
The expensive cost is quieter.
You paid for a cloud migration and you are still running an on-premises engineering culture. You are paying a cloud bill and maintaining Compose files, mocks, seed scripts, and a local environment that breaks for one developer every sprint for reasons nobody can reproduce. You have all the operational overhead you migrated to escape, plus the bill.
Your team’s expertise is accumulating in the wrong place. After two years, they are excellent at running Postgres and RabbitMQ locally and have never once dealt with a DynamoDB throughput exception, an SQS visibility timeout expiring mid-handler and redelivering the message, or an IAM role that works in Dev and not in Test. That knowledge only comes from using the thing, and every mock defers it.
And the moment when the mismatch surfaces is always the worst available moment. Not during development, when it’s a bug. During release, when it’s an incident — with an audience, and a rollback, and someone senior asking how this wasn’t caught.
The diagram was right the whole time. Nobody had run it.
We build cloud-native solutions. Let’s develop them that way.
]]>
Comments
Signing in with GitHub posts your comment to Discussions. Or just email me.