Cross-Language Transfer of Harm Probes in Vision-Language-Action Models

Research by Chris Shi · drafted with Codex from my experiments and manuscript.

In frozen robot policies, a harm probe can score well in English and near chance in Chinese. The VLM baselines retain much stronger cross-language detection.

01 · The English-to-Chinese gap

View full size

Swipe across the plot, or view it full size.

Every probe here is trained and tested within one frozen model. EN → ZH means fitting on English activations and evaluating on Chinese activations from that same model. Each model gets its own independently fitted probes. AUROC measures ranking: 0.5 is chance.

What should survive translation?

A useful internal harm detector should still rank a hazardous instruction above a safe one when the instructions arrive in another language. English validation alone cannot tell me whether it will.

Language-model research gave a reason to expect some transfer. Zhao and colleagues distinguish harmfulness representations from refusal representations. Aziz, Hanif, and Koto find that harmfulness directions from high-resource languages remain discriminative across 23 languages in three LLMs, even where refusal deteriorates. Those are results about language models; my question was whether an English harm readout would remain useful inside a robot policy. The VLM baselines provide the direct experimental context for that question.

I measured ranking with AUROC, rather than choosing a refusal threshold. An AUROC near 0.5 means the probe does little to order hazardous and safe test instructions correctly. Adjusting a threshold cannot repair that ranking.

Two ways to make the detection task

Safe examples determine what a probe has to distinguish. Balani and Panda show why negative-example difficulty matters for cross-language harmfulness probing in LLMs. I used two complementary designs:

Both sets cover six languages, with images held fixed across translations: 4,560 inputs in A; 5,760 in B. Instructions and translations were authored and audited with a language-model assistant. The audit flagged no meaning errors in the Chinese hazard texts, and every checkpoint used the same audited original version. Shared translation artifacts remain a limitation.

For every model and hidden-state index, I fitted an English logistic probe and tested it on held-out English groups and their translations. Five folds kept translations together, grouped by instruction item in A and frame in B. Chinese test items were translations of held-out groups, not of training items. A Chinese refit uses the corresponding Chinese training groups. Plots average folds and fixed deep layers; no best layer was selected.

English performance overstates cross-language usefulness

Start with Set B, the right panel of the first graph. Prismatic, MiniVLA VLM, and PaliGemma retain Chinese AUROCs of 0.92, 0.93, and 0.81. In the four VLAs, English-to-Chinese scores are 0.51, 0.57, 0.57, and 0.55, despite English scores of 0.96, 0.95, 0.87, and 0.75 for OpenVLA, MiniVLA Bridge, π0.5 base, and π0.5 DROID respectively.

The harder negatives in Set A lower English detection too. OpenVLA reaches 0.72 and MiniVLA Bridge 0.65 in English. But the cross-language contrast remains: the four VLA probes score 0.50, 0.51, 0.52, and 0.51 in Chinese, while the three VLM baselines score 0.73, 0.78, and 0.69.

Chinese is not an isolated failure. The next graph includes every non-English test language. On Set B, Japanese and Korean transfer stays between 0.51 and 0.56 across the four VLAs. Spanish retains partial transfer in OpenVLA (0.71) and MiniVLA Bridge (0.66).

02 · All five target languages

View full size

Swipe across the plot, or view it full size.

The depth profiles also help interpret the summary. In each VLA, the English readout remains stronger than its English-to-Chinese result across the deep layers. The gap is not an artifact of choosing one especially unfavorable layer.

03 · The gap across depth

View full size

Swipe across the plot, or view it full size.

What the controls change about the story

Set B is lexically easy. A difference-of-means direction on mean input-token embeddings already reaches 0.92–0.93 English AUROC in Prismatic, OpenVLA, and both MiniVLA models. So a strong deep English score alone does not establish a sophisticated scene-level hazard representation.

Even this simple embedding baseline transfers better to Chinese than the deep English logistic probe in OpenVLA (0.66 versus 0.51) and MiniVLA Bridge (0.71 versus 0.57). The estimators differ, so this is a useful baseline comparison rather than a controlled attribution of the gap to depth.

I also checked two narrower explanations. Length matching counts only hazardous–safe test pairs whose token counts differ by at most one. Holding out a hazard category asks whether a direction generalizes beyond categories used to fit it. These controls use difference-of-means directions, separately from the headline logistic probes.

The controls preserve stronger transfer in the two VLM baselines tested here, including when test pairs are matched for length or come from a held-out hazard category.

Chinese refits recover part of the missing discrimination

A failed English readout does not establish that Chinese hazardousness is unrecoverable. On the same Chinese test groups in Set B, fitting the probe in Chinese raises OpenVLA from 0.51 to 0.69, MiniVLA Bridge from 0.57 to 0.85, π0.5 base from 0.57 to 0.76, and π0.5 DROID from 0.55 to 0.66.

04 · Refit in Chinese; inspect the action expert

View full size

Swipe across the plot, or view it full size.

The recovery is smaller on Set A: Chinese refits reach 0.55, 0.59, 0.56, and 0.53 in the same four policies. That boundary matters. The results support both weaker separability on the harder task and poorer cross-language alignment for these readouts.

The right panel moves the measurement inside π0.5’s action expert. I read its first action position at the initial flow-matching time, t = 1, with identical Gaussian noise across examples. The expert attends to the full instruction-and-image prefix. In π0.5 base, the deep averages are 0.91 for English fit/test, 0.81 for Chinese fit/test, and 0.58 for English-to-Chinese transfer. The DROID expert has the same ordering: 0.81, 0.74, and 0.60.

So the language gap persists in the expert’s hidden states, alongside useful within-language detection. These measurements evaluate detector ranking in frozen activations.

The scope of an English safety result

A possible conclusion concerns the validity of a detector: high English AUROC does not establish that an English-trained harm probe will rank translated hazards correctly in a VLA. The VLM baselines show stronger transfer under the same two hazard designs. Chinese refits show recoverable signal that the English readout misses, with recovery depending on the task and readout.

For an activation-based detector, one could report source-language validation, cross-language testing, and a target-language refit on the same held-out groups. Together they distinguish a detector that works in English from one that generalizes, and a poor transferred readout from a representation that remains difficult to separate even with an in-language fit.

Close