There’s a famous engineering story — the kind every senior engineer has heard at least once — about a Pontiac owner who complained to the manufacturer about a very strange problem.

Whenever he bought vanilla ice cream, his car wouldn’t start when he got back to it.

Chocolate? Fine.

Strawberry? Fine.

Vanilla? Dead car. Every time.

Most companies would’ve told him to stop drinking on the job. Pontiac sent an engineer instead.

Turns out vanilla was the store’s most popular flavour, so it sat in its own freezer right at the front. Grab it and go: two minutes, tops.

Every other flavour was further inside, at a different counter. Longer walk, longer wait.

That extra time let the hot engine cool down. On the quick vanilla trip there wasn’t enough of it. The fuel in the lines was still hot, hot enough to turn into vapour and block the flow to the engine. This is called vapour lock.

The problem was never vanilla. It was the time between stopping the car and restarting it.

Vanilla didn’t cause anything. It just happened to be standing at the front of the shop.

(Real story or engineering legend, nobody’s fully sure. Doesn’t change the lesson.)

Three things we always mix up

When something breaks, we tend to mix three very different things into one.

1. Facts

What did we actually observe?

“When I buy vanilla, the car doesn’t start.”

That’s a fact. We can reproduce it, measure it, test it. Unlike the other two, it doesn’t change depending on who is in the room.

2. Feelings

How does it make us feel?

“This is scary. I just bought this car and now something’s wrong with it.”

Completely real, and completely useless as evidence. The worry is what makes people rush; it can’t tell you anything about the car.

3. Assumptions

What do we think is causing it?

“Something about vanilla is messing with the engine.”

A guess dressed up as an explanation. It might turn out to be right. It might turn out to be the thing you quietly stop mentioning.

The trouble starts the moment we treat the guess as a fact.

Engineering does this every single day

A production system goes down.

“It’s the database.”

“No, it’s the network.”

“It started right after the deploy — so it must be the deploy.”

Five minutes later, the team is arguing about solutions to a problem nobody has actually confirmed.

The only real fact on the table is usually something like:

“Requests started timing out around 10:35.”

And that is all we actually know. Database, network, deploy, traffic spike, cache, race condition: every one of those is still a guess, just delivered with confidence.

“We don’t know yet” is a perfectly good engineering answer. Pretending otherwise is how you spend four hours fixing something that was never broken.

Don’t debug the explanation. Debug the observation.

This is the whole point of the vanilla story, and the hardest habit to build, because the explanation is always more interesting than the observation.

Someone says:

“It only happens with customer X.”

Don’t rush to check customer X’s account like it’s cursed. Ask instead:

What’s actually different when we process customer X?

Bigger data? A different code path? A specific time of day? A setting only they use?

Customer X might simply be your vanilla ice cream. The pattern is real; the explanation might not be.

The engineer Pontiac sent didn’t investigate vanilla. He went shopping, three times, with a stopwatch. What he was really doing was looking for what co-varies with the failure, and the answer turned out to be elapsed time, which nobody had thought to write down because nobody thought it mattered.

The move is to take the thing that supposedly explains the failure and list everything that travels with it:

  • Volume. Is their dataset an order of magnitude bigger than everyone else’s?
  • Shape. Do they have a null, an empty array, a 500-character name, a timezone nobody else uses?
  • Path. Do they hit a different endpoint, region, tenant, or feature flag?
  • Timing. Do they run their batch at the same minute as something else of yours?
  • Concurrency. Are they the only customer with two users clicking at once?
  • History. Were they migrated from the old system, and does anyone still know what that migration did?

Now you have six testable statements instead of one unfalsifiable one. Any of them can be checked in an afternoon. “It’s customer X” can’t be checked at all, which is exactly why it survives so long in a chat thread.

The client everybody knew was cursed

I ran into this on a financial processing pipeline for short-term rentals. Transactions arrived on an SQS queue, and a Lambda function sat on that queue and drained it.

One client’s work took ages. Not failing — just crawling, for hours, while every other client went through in minutes.

And that client already had a reputation. They’d been a source of strange requests for months. Unrelated things broke on their account. So by the time this landed, “it’s them again” wasn’t a theory anyone had to argue for. It felt earned. It genuinely felt like a curse, and I believed it for longer than I’d like to admit.

That reputation was our vanilla ice cream. The pattern was completely real — it was always them — and the explanation was completely wrong.

What was actually different about them had nothing to do with who they were. It was how they submitted work. Every other client accumulated transactions per day, running totals rolling forward. This one submitted an entire year of them in one go, which hit the queue as a single enormous burst of messages.

So the load we had tested, repeatedly and carefully and many times over, covered every shape except that one. Small volumes: fine. Large volumes spread across days: fine. A year of financial transactions arriving all at once: never tested, because nobody had thought of it as a case that existed.

And that Lambda’s concurrency was capped at one. Not a decision anybody had made: it was whatever the function had been deployed with on day one, and nobody had revisited it since. One invocation at a time, every message waiting for the one in front of it.

And this is the part worth being precise about: nothing was slow. Every individual message was processed quickly, nowhere near any limit. There were simply an enormous number of them, and they went through the queue in single file. At normal volumes that was invisible, because one at a time was plenty and we never came close to the ceiling. With a year of transactions stacked up behind it, the serial approach was the problem.

When I finally sat down and worked through what actually co-varied with the slowness, it wasn’t the client. It was how many messages arrived at once, against how many could be worked on at once. One of those numbers was an order of magnitude past anything we had tested. The other was one.

I raised the concurrency from one to ten, so ten of them ran in parallel instead of one after another. That was the whole remedy, and because Lambda only runs what it’s given, normal operation cost us almost nothing: the extra capacity existed only when the queue was deep enough to need it.

Months of “that client is cursed,” and the answer was a concurrency setting nobody had ever looked at.

I’d like to report that the nickname retired along with the setting. It did not.

What’s uncomfortable in hindsight isn’t the mistake itself, it’s how reasonable we were being while we made it. Every previous failure on that account was real evidence — for the wrong conclusion.

The test that would have settled it in an hour: take a different client’s data, reshape it into a year in a single run, and push it through. If that crawls too, the client is innocent and the volume is the culprit. Nobody ran it, because nobody thought there was anything to test.

The test is the whole job

Converting a theory into a check is the part that separates debugging from arguing. It usually means finding the one measurement that can only come out one way.

“It must be the deploy — the timeouts started right after it.”

The deploy landed at 10:31. Timeouts began at 10:35. So did the hourly reconciliation job. Both are “right after.” Roll back the deploy and the timeouts should stop immediately; pause the job and they should stop at the top of the next hour. Whichever moves is your answer, and finding out takes about ten minutes. Arguing about which one feels more likely can comfortably fill an afternoon.

“It’s the database.”

Is the database slow, or is it starved of connections because something upstream stopped returning them to the pool? Those look identical from the application and want opposite fixes. Measure at the database, not at the thing complaining about the database.

“It only happens in production.”

Production is not a cause, it’s a bundle of differences: data volume, concurrency, config, network path, TLS termination, someone else’s traffic. Name them, then rule them out one at a time.

But it works on my computer

Which brings me to the phrase every team has heard, usually from the same person.

We had a developer who said “but it works on my computer” every single time a bug turned up in production. It became a running joke. Someone would open the incident channel and half the team was already typing it before he could.

Here’s the thing, though: he was right. Every time.

It did work on his computer. That isn’t a defence, it’s a measurement — and a genuinely useful one. It tells you the code is capable of working, so whatever is broken lives in the difference between his machine and production. It’s the most valuable sentence anyone can offer in the first five minutes of an incident, and we turned it into a punchline.

“It works on my computer” is exactly “chocolate is fine.” And chocolate was fine. That was the clue. Nobody laughed at the Pontiac owner for mentioning it; an engineer wrote it down and went shopping.

What we should have done, every time he said it, was ask the follow-up: then what’s different about yours? Java version, a stale local config, a database with fifty rows instead of fifty million, no concurrent traffic, a mock standing in for something real, a timezone set to his own.

Most of those differences are accidents. Some are designed in, which is worse.

The joke got a laugh. The question would have got us an answer.

What the guessing actually costs

The first wrong theory is cheap: you lose an afternoon. The habit is what gets expensive, because wrong explanations don’t stay in the incident channel. They get written down.

They become the runbook step everyone follows for the next two years. They become the alert that fires on the symptom you misdiagnosed. They become the retry someone added around the wrong call, the cache in front of a database that was never the problem, the nightly restart nobody dares remove because it “fixed” something in 2023.

Every one of those is a defence built against the vanilla.

And because the real cause was never found, it comes back. Different symptom, different quarter, different on-call engineer — who now has a runbook confidently pointing in the wrong direction, and a system carrying three years of scar tissue from the last four times this happened.

Nobody on the team experiences this as failure. It just feels like working on a hard system that breaks a lot.

A simple rule

Separate, every time:

FACT — What do we know happened?

FEELING — What worry does it create?

ASSUMPTION — What do we think caused it?

Then do the one thing most people skip: test the assumption.

Don’t argue about whether it’s the database, the network, the deploy, or the vendor. Build a small test that can actually prove or kill the theory.

That’s debugging. Everything before it is opinion with confidence attached.

Good engineers don’t guess the fastest — they test the hardest

They can turn:

“I think it’s the database.”

into:

“Let’s test if the database is actually responsible.”

And then — the hard part — accept it when they’re wrong. The goal was never to be right; it was to find out what’s actually happening.

This is really what troubleshooting culture is about.

Not finding someone to blame. Not being the person with the fastest answer. It’s about building a habit, as a team, of separating facts from guesses and testing before deciding. Teams that do this learn faster. Teams that don’t will keep repeating the same guesses year after year, and calling it experience.

Whether or not the Pontiac story is 100% true, it points to something that is: the pattern was real, the explanation was wrong. Worth remembering next time your production system does something that makes no sense.

Don’t blame the vanilla. Find out what’s actually different about it.