This morning my car flinched at nothing.
I was on a quiet two-lane road, FSD engaged, doing about 26. Ahead of me was a dark streak running down the pavement — the kind of mark you'd assume was a spill. The car eased left to go around it, tracking the edge of the streak like it was a curb. Smooth, confident, deliberate. Then I got close enough to see what it actually was: a dried, light-colored stain. Not a puddle. Not oil. Nothing with any physics to respect at all. Just a discoloration on dry asphalt.
The car didn't avoid a hazard. It avoided a pattern that resembled one.
I've seen this before, and so have you if you drive one of these. Last fall, owners were posting clips of their cars swerving around leaf piles. Shadows that read as potholes. Tar seams treated like lane lines. Tesla even has a name for the polite version of it — the June software notes list "mitigated unnecessary lane biasing" as a fix, right next to "add pothole avoidance." Which tells you everything: the exact sensitivity that dodges a real pothole is the sensitivity that swerves around a harmless smear. It's one behavior wearing two coats.
Here's why it happens, and why it matters far beyond one car.
Photons in, controls out
Tesla threw away the rulebook. Older self-driving stacks ran on explicit logic — if red light, then stop — hundreds of thousands of lines of it. The current system deleted almost all of that and replaced it with a single neural network trained on millions of clips of humans driving. Raw camera pixels go in one end; steering, brake, and accelerator come out the other. Nobody wrote a rule for stains. Nobody wrote a rule for anything. The network simply learned the statistical relationship between what a scene looks like and what a human did next.
And humans edge away from dark streaks on the road. We do it without thinking. So the network learned to do it too — including the part where we're sometimes wrong. It inherited our instincts, reflexes and superstitions bundled together, because it can't tell which is which. It has no model of hazard. It has a model of what hazards tend to look like, which is not the same thing, and the gap between those two is exactly where my car drifted this morning.
The newer versions aren't pure imitation anymore — Tesla now layers reinforcement learning on top, rewarding the behaviors it wants and penalizing the twitchy ones, using hard examples pulled from the fleet. That's the missing half of the picture, and it's oddly reassuring: the flinch isn't permanent. The "unnecessary lane biasing" they're sanding down and the "pothole avoidance" they're promising are the same instinct being retrained rather than deleted — the system taught, slowly, when to trust itself, one rewarded example at a time. But retraining a reflex is not the same as installing understanding. It shifts where the line sits between "swerve" and "ignore." It doesn't give the car any idea of what a stain actually is.
The same failure, in a different body
Last month the Wall Street Journal ran a piece on home security cameras doing the identical thing — just with words instead of a steering wheel. A woman in Houston got a push alert that her neighbor's house was on fire. The "flames" were a car's brake lights in the dark. Another camera turned a window reflection into a tornado warning. Across the story: a raccoon flagged as a bear, a corgi logged as a pig, a woman in brown athleisure sweeping her lanai identified as a brown bear in the yard.
David Doermann, a computer-vision professor quoted in the piece, put it about as cleanly as it can be put: these systems are remarkable at recognizing visual patterns but essentially have no common sense. Impressive one moment, completely wrong the next.
That's not a camera problem or a car problem. It's a perception-without-comprehension problem, and right now it's everywhere vision AI ships. The eyes are superhuman. The understanding behind them is close to zero. The system sees the what-it-looks-like flawlessly and the what-it-is not at all.
I work on this class of product, so I'll say the uncomfortable part plainly: the phantom fire and the phantom swerve are not bugs we'll patch away one by one. They're the tax you pay for pattern-matching without a layer that knows what things mean. And that tax doesn't stay abstract. It shows up as the customer who stops trusting the alert, then stops opening the app, then stops paying. A false alarm is a small thing. A thousand of them is a churn number.
What actually closes the gap
More pixels won't fix it — the cameras already see fine. What's missing is judgment, and that comes from three things stacking on top of raw perception:
- Confidence gating — the system knowing when it doesn't know, and staying quiet or cautious instead of confidently narrating a house fire.
- Temporal reasoning — using time, not a single frame. A fire persists and spreads; brake lights don't. A pothole stays put across frames; a shadow moves with the sun.
- A comprehension layer — a model that carries enough world-knowledge to check its own output against meaning. A stain is flat. A raccoon is not a bear. A reflection is not a person.
None of this is exotic. It's the arc the whole field is walking right now, from "describe the pixels" toward "understand the scene." We're early. The honest version — the one even the vendors give — is that there are billions of real-world scenarios these models still have to learn before they stop being surprising.
But I keep coming back to that stain this morning. My car saw it perfectly and understood it not at all. Until we close that specific gap, we don't have intelligent machines. We have extraordinary eyes attached to nothing that knows what it's looking at — and a lot of very confident guesses in between.