Perceives but Doesn’t Brake

A reasoning driving model sees the stopped car and names it in its own reasoning trace. Its planned trajectory drives into the car anyway. The hazard is encoded inside the model, but it has almost no say over how far the plan goes. One added term in the frozen sampler gives it that say back, with no retraining.

Sampled plans entering the obstacleCARLA hazards, seen sites: 58.1%14.5%
Closed-loop rollouts in contact24 must-brake scenes: 130
How closely reach follows obstacle rangeSpearman ρ, 273 logged states: +0.01+0.68

Figure 1 of the paper. Top: a camera image with a cyclist ahead feeds a visual representation and a reasoning trace that says brake for the cyclist ahead; both connect to the action expert only weakly, while ego history connects strongly, and the plan collides. Bottom: a hazard energy steers the trajectory during generation, from noise at t = 0 to a clean, safe trajectory at t = 1. — Visual hazard evidence and the model’s reasoning trace have much weaker authority over the longitudinal plan than ego history. The intervention adds a hazard-conditioned energy gradient during flow integration and leaves the model weights untouched.

Abstract

Reasoning-augmented vision-language-action (VLA) driving models can perceive and reason about safety-critical hazards yet fail to translate that evidence into safe actions. We characterize this failure as a perception–action binding gap: hazard information is available inside the model but exerts insufficient control over trajectory generation. Using matched simulated counterfactuals and logged driving replay, we show that in-path hazards are actually encoded in the visual representation, but longitudinal plans are dominated by ego-speed history and respond only weakly to hazard proximity or the model’s own reasoning trace. Further analysis shows that the trajectory generator is capable of braking but sometimes fails to do so due to the perception-action binding gap. Without retraining or modifying any weights, we inject a hazard-conditioned energy into the frozen flow-matching sampler of Alpamayo-1.5, restoring the missing coupling at generation time. With exact obstacle geometry, guidance cuts footprint intrusion from 58.1% to 14.5% and recovers safe braking whenever sufficient stopping distance remains, eliminating collisions entirely in our braking-relevant closed-loop battery. It also reverses the diagnosed dependence of the plan, from tracking ego speed to responding to obstacle range. These results separate hazard perception from hazard control in driving VLAs, show that unsafe behavior can arise even when the model has both the evidence and the ability to act on it, and propose a simple fix for bridging this binding gap.

Accompanying video · 2:20

It carries no voice-over; the captions are the narration.

01The failure

The model names the stopped car in its reasoning, then plans straight through it.

We study one safety-critical state: a moving ego vehicle approaching a stationary obstacle in its lane. Every instrument below pairs a hazard scene with an identical twin in which only the hazard differs, so any change in the plan is caused by the hazard and not by the site.

58.1%

of sampled plans enter the obstacle’s footprint on designed CARLA hazards. At sites held out in geometry and appearance: 65.0%.

Open loop · 144 hazard scenes · K = 6 samples each

+54.2pts

front-collision rate once a frozen car is placed 40 m ahead, from about one rollout in four to four in five.

Closed loop · 24 paired reconstructions · 95% CI +33.3 to +75.0

13of 24

must-brake scenes end in contact with a lead car that stops where the log places it. The model slows by about 2 m/s, which is not enough.

Closed loop · logged lead vehicle · no inserted actors

Matched closed-loop rollouts

Two rollouts of the same reconstruction, replanned at 10 Hz. Left: the lead car keeps driving (the matched control). Right: it stops. Nothing else differs, so the collision on the right is caused by the hazard. Across all 38 dynamic scenes minimum clearance drops by a paired 23.7 m and improves in none.

02Diagnosis

Of three possible causes of an unsafe plan, only one fits the evidence.

The hazard may be missing from the representation (availability). The trajectory generator may be unable to produce a stop (action-side sufficiency). Or the evidence is there and a stop is expressible, but changing the evidence barely changes the plan (action authority). Standard driving metrics conflate the three. We separate them with interventions that change one input while holding the others fixed.

The evidence is present and a stop is expressible, yet the evidence has little authority over the plan. That combination is what we call a perception-action binding gap. Wire widths are drawn to the measured authority of each input.

Perceives

Doesn’t brake

The hazard is decodable at every range. Up to 12 m, the plan still enters it.

Matched CARLA pairs, fraction of scenes per obstacle range (majority of K = 6 samples). All 168 scenes at 8 to 12 m are decodable and intrude. At 8 m, 86% of scenes plan a near-stop (forward reach under 8 m) and still enter the footprint: the braking comes too late. Whiskers are 95% CIs, bootstrapped over sites.

Data

What the plan listens to

Shift of the mean plan (RMS, metres, log scale) when one input is removed or swapped, on 120 logged clips at the native four-camera input. Along the road, ego history moves the plan about 19 times more than all four cameras together. Across the road the two are level: vision does steer the car; it has little say over its speed. On 120 clips that NVIDIA flags as long-tail events the longitudinal ratio is still about 12. Whiskers are 95% CIs over clips.

Data

Rotating a stopped car broadside at 8 m changes what the trace says far more than what the plan does

Reasoning traces that demand a stop

10.4% → 99.0%

Planned forward reach, against the 8 m to the obstacle

moves +0.41 m

The car is rotated from lane-aligned to broadside, holding site, actor, ego motion and camera input fixed. Almost every trace now calls for a stop; the plan barely changes. Rotation also changes the image, so we read this as corroborating evidence. The cleaner test, swapping traces between 120 logged clips, moves the plan by only 0.44 m.

03The fix

Adding one term to the frozen sampler restores braking without retraining.

Alpamayo-1.5 draws its trajectory from a flow-matching expert in ten integration steps. At every step we subtract the gradient of a stopping energy E, so a safety signal s acts on the plan while it is being generated. Weights, inputs, reasoning trace and initial noise stay as they were.

x_{k+1} = x_k + Δt v_θ(x_k, t_k; c) − η_k ∇_x E(x_k; s)

Flow-matching sampler

x_0 ~ N(0, I)
for k = 0, …, 9 do
t_k ← k/10
x_{k+1} ← x_k + Δt v_θ(x_k, t_k; c)
end for
return x_10

With energy guidance

x_0 ~ N(0, I)
for k = 0, …, 9 do
t_k ← k/10
η_k ← η_0 sin(π t_k)
x_{k+1} ← x_k + Δt v_θ(x_k, t_k; c) − η_k ∇_x E(x_k; s)
end for
return x_10

Stop position: Penalizes forward travel beyond a safe stopping point d_safe in front of the obstacle. This term does the braking.
Speed cap: Limits excessive acceleration and softly penalizes reversing. Its target is never below the current speed, so this term never asks for braking.
Jerk: Keeps the speed profile smooth.

The safety signal s = (g, d, l) holds an activation gate, an obstacle-centre range and a half-length, which set d_safe = max(d − l − L_ego − b, 1). The gradient runs through the trajectory’s kinematics and not back through the network, so it adds 3.8 ms to a 2.30 s replan (0.2%) at the native K = 6 samples.

The sampler, step by step

This is a toy model for intuition: an analytic flow stands in for Alpamayo’s expert, while the energy, its schedule and its constants are the paper’s.

t = 1.00

Obstacle ahead 20 m Ego speed 8 m/s

Footprint intrusion: 6 / 6
Mean forward reach: –
Safe stopping point: –
Shortest possible stop: –

Six plans integrate from shared noise (t = 0) to a plan (t = 1). Without guidance their reach follows ego speed and hardly moves with the obstacle; drag both sliders to see it. With guidance the same noise ends short of the obstacle, and reach follows the obstacle’s range. When the shortest possible stop is longer than the room left, no plan avoids contact. In the paper, all of the remaining intrusion at seen sites sits at 8 m, where a car or truck leaves no stopping distance.

Given the exact obstacle geometry, the frozen generator brakes.

Designed CARLA hazards at seen sites (n = 144) and at six new sites held out in geometry and appearance (n = 91). Near-stop means forward reach under 8 m. At seen sites the remaining intrusion sits entirely in the 8 m range bin, where a car or truck leaves no stopping distance; it is 0.0% at 12, 20 and 30 m. Whiskers are 95% CIs, bootstrapped over sites.

Data

Six object kinds, same four-camera input

Left, the frozen model: its six samples run into the obstacle 12 m ahead. Right, the same model and the same noise with stop guidance: none do. The plot underneath is planned speed along the plan.

The plan’s dependence flips, from ego speed to obstacle range.

Held-out logged US scenes at the native four-camera input: 273 states from 185 clips, each with the ego moving at 4 m/s or more and a stationary object 8 to 45 m ahead in its corridor. Each dot is one state’s planned forward reach, averaged over K = 6 samples. Without guidance, reach follows logged ego speed and not obstacle range. With exact-geometry guidance it follows range, and speed matters much less. ρ is Spearman’s rank correlation with a 95% CI clustered by clip. These are nominal scenes, mostly vehicles parked at the curb, so they measure what the plan tracks rather than how it handles a hazard.

Data

04Closed loop

With guidance, every must-brake rollout stops short of the car, even while replanning at 10 Hz.

An open-loop score judges one plan against a frozen scene, and the model might recover once it can watch the gap close, but it does not. In AlpaSim we take a lead vehicle that the reconstruction already contains and let it either drive on or stop at a pose the log places it. In the 24 scenes where avoiding it requires braking, the frozen model hits the stopped car in 13, and the guided model in none.

Minimum clearance to the stopped car, per scene

13 / 24 → 0 / 24rollouts in contact

−0.02 m → 18.08 mmedian minimum clearance

24 / 24scenes improve, paired Δ +17.3 m (95% CI 15.0 to 19.6)

Each row is one scene, and further right means more distance kept. The orange dot is the closest the frozen model came to the stopped car, the green dot is the same scene with guidance, and the gray line between them is what guidance adds. Filled dots sit in the shaded band, where clearance is 0 m or less and the two cars touch. The blue tick is the matched control, the same scene with the lead car driving on. Rows are sorted by the frozen model’s clearance; hover one for its numbers. These are the 24 must-brake scenes among 38: the other 14 never test braking, because the model’s own path already clears the stopped car (11) or the control never reaches it (3). Contact means the ego footprint overlaps the stopped car’s. The gate is fixed on (g = 1) and the constraint is accurately specified, so this tests whether the generator can enforce a correct stop. When to switch the constraint on is the subject of section 05.

Data

Closed loop, twice real time

The lead car stops in both rollouts. Left: the frozen model. Right: the same model with stop guidance, coming to rest behind the car. The speed trace underneath marks the moment of contact.

Dynamic closed-loop test

Scenes	The lead car	Contact ↓	Clearance ↑	Δ clearance
All 38	drives on	0.0%	24.31 m	ref.
All 38	stops	42.1%	0.09 m	−23.7 m
Must-brake 24	drives on	0.0%	24.31 m	ref.
	stops	54.2%	−0.02 m	−23.1 m
	stops, + guidance	0.0%	18.08 m	+17.3 m

Clearance is the median minimum separation. Δ is paired against the control; the guided row is paired against the frozen model with the car stopping, so it is the effect of the intervention. 95% CIs, scene-clustered bootstrap: contact 42.1% (26.3 to 57.9) and 54.2% (33.3 to 75.0); Δ −23.7 m (−29.0 to −18.5), −23.1 m (−28.6 to −17.5) and +17.3 m (15.0 to 19.6).

A second, static test re-poses a reconstructed car 40 m ahead and freezes it. Front collisions rise by 54.2 points across 24 paired scenes; of the 18 scenes the frozen model drives cleanly unedited, 13 collide once the car is added. A replay of the unedited policy on the same scenes shows no comparable shift.

05Readouts

Much of the missing constraint is already inside the model.

The oracle told the sampler when to brake (the gate is on) and where (exact geometry). Here both come from small readouts of the frozen vision tokens: a logistic hazard-presence gate and a ridge range estimate, fit on clips disjoint from evaluation. The energy is unchanged; only the source of s differs.

Footprint intrusion at sites held out in geometry and appearance

n = 91 hazard scenes at six new sites. Each step changes only where the gate and range come from. Pooling the token grid into mean, maximum and standard deviation discards the layout a distance estimate needs: under appearance shift that readout overestimates near obstacles, so real hazards slip past the gate. Adding the front-tele camera helps. A small attention readout over the same tokens helps more, cutting range error from 3.28 to 2.21 m with no measured reach loss on clear scenes. Whiskers are 95% CIs bootstrapped over sites; the attention bar shows ±1 SD over five seeds.

Data

Readout quality

Sites	Readout	MAE	Recall	Intrusion
Seen	pooled, front-wide	2.20 m	81%	13.4%
	+ front-tele	1.23 m	74%	16.1%
	attention	1.99 m	76%	16.3%
Unseen	pooled, front-wide	3.28 m	55%	28.6%
	+ front-tele	2.43 m	66%	18.5%
	attention	2.21 m	81%	11.7%

Seen sites are evaluated leave-one-site-out (n = 144); unseen sites are new in both geometry and appearance (n = 91). Attention rows average five seeds.

Can the reasoning trace decide when to brake?

Gate	Recall ↑	False stop ↓
Vision readout	71% (62 to 80)	11% (6 to 16)
Reasoning trace	55% (45 to 65)	15% (10 to 21)
Either	87% (80 to 93)	18% (12 to 25)

On a separate held-out routing corpus, a trace that says to slow or stop catches hazards the vision readout misses, at the price of more false stops. Against the stronger presence readout used by the guidance arm it adds only 0.9 recall points for 16.7 points of false flags, and it gives no metric range.

06Scope

Other driving policies show related failures.

Each model is tested through its own native interface. They differ in training data, sensing, action space, language use and ego-state conditioning, so these comparisons bound the phenomenon rather than establish a shared cause.

Senna · language-level decisions

12%hazard12%clear

Senna predicts a driving decision in words, so we compare its decelerate or stop rate on matched hazard and clear scenes. The rate is 12% in both, although its output often describes the stopped vehicle. Changing the supplied ego-speed token does change how often it brakes.

VaVAM · video to trajectory, no ego state

Samecollision rate in closed loop

On matched logged scenes VaVAM’s footprint intrusion is much lower than Alpamayo-1.5’s. In closed loop the two collide on the same scenes at the same rate, and VaVAM leaves the drivable surface in most rollouts. It reacts sideways, and a footprint metric can count an inadmissible swerve as avoidance. Fed one frame repeated instead of a clip, its intrusion drops to zero only because the plans become too short to reach the obstacle.

07Evaluation

Why logged evaluation misses it, and what we did not show

A binding gap is a failure of conditional response: adding a hazard, with everything else held fixed, should change the action, and here it barely does. Average prediction quality can miss that, and logged data cannot isolate it, because hazard presence, ego motion, scene geometry and the driver’s behavior all vary together. The state itself is also rare: human drivers avoid it. Under our strict filter nuScenes contains only a handful of frames in which a moving ego vehicle is still closing on a stationary in-path obstacle at speed.

We therefore use simulation to supply the counterfactual, not to estimate how often real collisions happen.

Every hazard we measure is synthetic, rendered in CARLA or re-posed in a NuRec reconstruction. Logged driving enters only as a coupling corpus, because the must-brake state is close to absent from it.
Footprint intrusion is an open-loop planning metric, not a crash probability. That is why the failure is also checked in closed-loop replay, with matched A/A replays as a control.
The model-derived gate holds across held-out CARLA sites but not across the shift from CARLA to NuRec.
A probe shows that hazard information can be recovered from the representation. It does not show that the policy uses it.
Related failures in other models do not establish that they share a cause.

08Citation

Cite the paper

BibTeX

@misc{cercola2026perceivesdoesntbrakeperceptionaction,
      title={Perceives but Doesn't Brake: A Perception-Action Binding Gap in Reasoning Driving VLAs},
      author={Matteo Cercola and Katie Z Luo and Simone Formentin and Marco Pavone},
      year={2026},
      eprint={2610.28184},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.28184},
}