Whether proprioception hurts a manipulation policy depends on what the policy can see

Research object

1. The contradiction

Recent work disagrees about proprioception:

• State-free policies are more robust. π0, ACT and Diffusion Policy without state lose far less under height, horizontal and embodiment shifts: 37–98 pp less (2509.18644).
• State rescues camera faults. π0.5 and GR00T with state keep 11–69 pp more success when a camera is blacked out or the robot is masked in the image (Blackout vs Freeze, 2609.39145).
• In distribution it barely matters. Most papers report differences within a few points.

The standard explanation is a shortcut. During training, state predicts the next action so well that the policy leans on it and attends less to the images (de Haan et al. 2019; Wen et al. 2020). That explains the harms but not the rescues.

Question. Which variable decides the sign of a state ablation?

2. Meta-analysis

2.1 Retrieval and coding

• Seeds. The three conflicting papers (2509.18644, 2608.03052, 2609.02546) and the two shortcut papers (1905.11979, 2010.14876).
• Graph walk. Both directions of the citation graph, plus a second hop from five first-hop papers that run state ablations. Semantic Scholar returned 912 papers.

• Screen. A high-recall title/abstract screen flagged 82. With 7 classics added, all 89 were read:

  • 49 excluded (driving, locomotion, Atari, datasets, no state arm);
  • 2 duplicates;
  • 3 unavailable.
• Coding. 14 papers give 123 numeric with-vs-without-state settings. One row per setting, with source table or figure: evidence_map/data/results.jsonl.
• Variables coded. Shift type, state frame, fusion, velocity terms, history length, action space, wrist camera, platform, backbone, pretraining and masking protocol.

2.2 Pre-set test

Each record is classed as:

• hurt: below −3 pp;
• null: within ±3 pp;
• help: above +3 pp.

Each paper carries a total weight of 1. A variable explains the contradiction if both hold:

• its levels predict the weighted-majority class with paper-weighted accuracy of at least 0.90;
• no level has counterexamples from two or more papers.

No variable passes. The best single variable is the type of evaluation shift (0.78). The best pair, shift × state frame, reaches 0.82. The failing level is the in-distribution one; every shifted cell is nearly clean:

2.3 Planner

Seven mutually exclusive hypotheses were scored with a Bayesian evidence planner:

Leaving out any one paper keeps H7 at 0.55–0.68. H6 and H7 were written after reading the papers, so the posterior ranks explanations and does not confirm them.

The most informative missing cells are configuration shift × corrected encoding. No paper tests one policy under both a configuration shift and a camera fault.

2.4 What the literature cannot separate

"Configuration shift" pools two different things:

• the robot starting from a new joint configuration;
• the objects being somewhere new.

The published harms under object shifts come from policies whose visual grounding is unknown. The published rows also never report how the no-state arm itself fared relative to its in-distribution score, beyond its success under shift. §4 shows that this missing number decides the sign.

3. Simulation setup

Simulator. LIBERO (MuJoCo 3.2.3, robosuite 1.4.1), CPU only.

Shifts. Every policy runs the same paired instances: 10 tasks × 5 init states per suite.

State arms.

• N: none.
• R: raw [eef_pos, eef_axis_angle, gripper_qpos] in the world frame.
• C: episode-relative, pos − pos₀ and axis_angle(R₀ᵀR), gripper absolute.
• **D** (pixels only): as R, but on 30% of training samples the state input is replaced by its training mean (state dropout, the regularised encoding of the pilot design). At evaluation the state is always given.

Policy and training. Every policy gets the task as a one-hot input and predicts a 10-step action chunk, of which 5 steps are executed. Demonstrations are no-op filtered.

Perception levels.

Pixel alignment. At evaluation, images are rendered at 128×128 and resized exactly as in training. Rendered frames match the dataset frames: mean absolute error 3–13 out of 255 when demo states are replayed (results/pix/render_alignment_spatial_t0.png).

Statistics. Differences are paired on identical instances:

• the effect is the mean over seeds;
• the 95% CI is a task-cluster bootstrap with each instance carrying all its seeds;
• the seed spread is the per-seed effects and their SD.

4. Results

All numbers are success rates in %, as the mean over 3 seeds, with per-seed values in brackets unless noted.

4.1 In distribution, LIBERO barely tests perception

• A policy that never sees an image solves 74% of in-distribution instances across the four suites.
• Moving the object 6 cm takes every blind policy to about 4%. So 6 cm is the smallest layout shift we tested that requires perception, and 3 cm does not (30–44% blind, single seed).

This agrees with LIBERO-PRO (2510.03827), which finds trajectories unchanged when the object is removed, and with MINERVA (2609.03715), where a task-id permutation drops a 0.54M CNN to chance. It sharpens them to a number with no images at all (research/novelty_check.md).

4.2 The effect of state, by perception level and shift

Paired differences in pp, with 95% CI and per-seed values:

Pixel success rates (N / R / C / D):

Seed spread. The per-seed SD of a paired effect was 1–8 pp across all experiments. Every effect whose CI excludes zero has the same sign in all three seeds, except two:

• ego layout R − N (+11/0/+9);
• pixel robot-init R − N (−2/+1/+13), whose CI includes zero.

4.3 Why the robot-start harm disappears with pixels

Drop from in-distribution to robot_init 0.1 (pp):

• The state arms degrade about equally at both perception levels (23–30 pp).

• What changes is the no-state arm.

  • The ego-oracle N arm sees only gripper-relative object offsets. Those carry no information about the arm's joint configuration, so the shifted start barely affects it.
  • The pixel N arm sees the whole arm in the agent-view image. It is exactly as start-sensitive as the arms given state.
• So "state hurts under robot-start shift" measures the no-state arm's invariance, not a harm of state per se.

Input-ablation probe. On 2,000 training frames, we measured how much the normalised action chunk changes (RMS) when one input is replaced by its training mean (state) or by zeros (images). Values are relative to the action standard deviation; files results/pix/reliance.json and results/seeds/ego_reliance.json.

• The idealised policies weight state about as much as their perception.
• The pixel policies weight state about half as much as either camera.
• With pixels, adding state does not change the imitation loss: final training MSE was 0.200 / 0.199 / 0.197 for N / R / C. With the ego oracle, state cut the loss 2.5×.

State becomes redundant once the images already show the arm, and redundant state is not a strong shortcut.

4.4 The relative frame: help, harm or nothing, depending on perception

C − R under robot_init 0.1, by perception level:

• Blind. The relative state starts at zero in every episode, so replaying from it avoids extrapolating an absolute state the policy never saw.
• Allocentric. World-frame objects need the absolute arm position to compute a reach. The relative frame throws that position away.
• Ego. Gripper-relative objects already carry the reach. The frame adds nothing.
• Pixels. The frame helps, which is the H7 pattern.

So "does a corrected frame fix state?" has no perception-free answer.

4.5 State dropout (D)

Paired differences for the pixel policies, 3 seeds:

• Under both shifts, D is indistinguishable from raw state, and it is worse than relative state in all three seeds.
• In distribution, dropout adds a small, consistent gain over raw state (+2.7 pp; +2 to +3 in every seed).

• Reliance probe. Replacing D's state with its training mean changes its actions by only 0.07 action-SD, about a fifth of R's 0.38–0.41. Dropout taught the policy to treat state as optional, and it then mostly ignored it.

  • Caveat: the probe's replacement value is exactly the dropout value D saw in training. The probe therefore measures how far D's actions depend on the state when it is present, not how it reacts to an unusual state.
• The question dropout was meant to answer remains open. Does reducing reliance on state give up state's camera-fault benefit (the pilot's trade-off T)? It cannot be answered here, because every pixel arm, D included, scores 0% under scene blackout (§4.6).

4.6 Camera faults: not reproduced

With the scene camera blacked out, all pixel policies scored 0%. The oracle analogue also scored 0%. Our small CNNs cannot do these tasks from the wrist camera alone, so this study says nothing about state's camera-fault benefit. That half of the evidence still rests on one paper (Blackout vs Freeze).

5. Interpretation

1. The sign of a state ablation is a comparison between two policies. It depends as much on what the no-state policy can perceive as on what state adds.

   • Under a robot-start shift, state hurts exactly when the no-state policy's perception is invariant to the arm's configuration: blind replay or gripper-frame objects.
   • When the cameras show the arm, which is the standard third-person LIBERO setup, both policies inherit the same sensitivity, and raw state is neutral.
2. Object shifts behave differently from robot shifts. With perception good enough to locate the object, state did not hurt at 6 cm; in the ego oracle it helped. A state harm under object shift in a real VLA would therefore indicate vision suppression: imperfect visual grounding that state lets the policy skip.

3. Regularising state away is not the same as correcting its frame.

   • With pixels, dropout made the policy largely ignore state and left its robustness unchanged (D ≈ R ≈ N under both shifts).
   • Re-expressing state relative to the episode start kept state in use and improved robustness to the robot-start shift (C − R +8 pp).
   • So when state does help, it helps through a frame that stays in distribution, not through less reliance.
4. For the planner's H6 vs H7. The pixel results favour an encoding effect under robot-start shift: C beats both R and N. But they do not reproduce the premise that raw state hurts under configuration shift. The pooled "configuration shift" cell should be split, and H6/H7 re-scored per sub-type, before any decision.

6. Testable predictions for large VLAs

Setup. One backbone (π0.5 from pi05_base), fine-tuned on LIBERO three times with identical data, cameras and recipe:

• N: stock pi05_libero, discrete_state_input=False;
• R: raw state as discretised prompt tokens;
• C: episode-relative state, renormalised;
• D: raw state with 30% state-token dropout during training (the pilot's planned extension arm).

The protocol, configs and scripts are in research/pilot_3x2_liberoplus.md and pilot/stage0/. Instances are paired, and the shifts are those of §3, at LIBERO-Plus magnitudes.

P2, P5 and P7 are the decisive ones. P7 is a controlled version of our explanation, and it can be run with the same checkpoints by changing only the camera setup at fine-tuning.

What would most change our conclusions. If P2 fails, state harms the policy in a way our small CNNs do not capture, such as a stronger state shortcut at VLA scale or with 256-bin state tokens. That would make the pilot's §3a decision table apply unchanged.

7. Limitations

• Small policies. At most 1.7M parameters, trained from scratch on 2 of 4 suites (Spatial, Object), at 64×64. π0.5 has a pretrained 2B vision-language backbone, state as discrete tokens, and different failure modes. The simulation bounds mechanisms; it does not predict π0.5's numbers.
• One robot-start magnitude with pixels (0.1). The ego oracle also covered 0.2 and showed the same sign, with 1 seed.
• Camera faults could not be tested (§4.6). As a result, the trade-off that state dropout was meant to probe (robustness bought by giving up the camera-fault benefit) is untested.
• One dropout rate (30%) was tried, and dropout was applied to the raw-state arm only.
• The allocentric oracle has 1 seed.
• Blind seed-0 layout rows were re-run after a reset fix. The seed-0 ID and robot-start rows predate it but are not affected by it.
• Meta-analysis coding was done by one reader (Claude) without a second human coder. H6 and H7 are post hoc.

• Novelty. We searched arXiv up to mid-October 2026 (research/novelty_check.md). The blind-LIBERO result extends known memorisation findings. We found no prior report of:

  • the robot-start versus object-shift reversal;
  • the perception-dependence of the frame effect.

Reproducing