The Piggyback Hypothesis: explaining and mitigating emergent misalignment
Why does finetuning a language model on narrow, misaligned examples make it broadly misbehave on totally unrelated questions? We trace the effect to a surprisingly small surface — the chat-template tokens — and show that a simple training-time regularizer largely fixes it.
TL;DR
Finetuning on a narrow domain of bad advice (e.g. wrong financial tips) makes models broadly unsafe — emergent misalignment (EM). We provide evidence that this happens because the shared chat-template prefix absorbs a query-independent “misalignment bias” during training, then piggybacks that bias onto every new query. Patching the prefix’s KV-cache back to its unfinetuned state restores alignment. Regularizing the prefix during training (TReFT) prevents the problem in the first place, beating data interleaving, and reducing off-topic generalization across abstention, tool use, and refusal by 54.3% on average.
A prefix you never look at may carry the training behavior
Every chat prompt is wrapped in a fixed template. The model sees the same prefix before every input. We hypothesize finetuning can potentially bind new behavior to that prefix, not to the query semantics.
After finetuning on misaligned examples in a narrow domain, the chat-template prefix (blue) ends up encoding a bias for misalignment. Because the same prefix is shared across all inputs, it piggybacks the misalignment onto unrelated queries. Replacing its KV-cache with the unfinetuned model’s recovers alignment. Regularizing it during training prevents the problem.
Key findings
EM is brittle to the input tokens
Replacing a handful of prefix tokens with near-neighbor embeddings raises alignment score. Doing the same to the user query — the part that should drive behavior — barely moves the needle on average. The misalignment seems to live in the prefix tokens, not the content.
Figure 2. Alignment scores after replacing tokens with ones of similar embeddings in different input segments. Parentheses show the change relative to each EM model’s alignment score before intervention (39.7 for Qwen-2.5-7B and 40.8 for LLaMA-3.1-8B). Prefix replacement consistently gives the largest gain.
Figure 3. Example on the finetuned EM model (Llama-3.1-8B). Subtle changes on the template prefix tokens can substantially alter model behavior. The user query (in orange) remains unchanged; only the prefix is perturbed (highlighted in blue).
Patching prefix representations causally restores alignment
We copy the prefix-token KV-cache from the unfinetuned model into the misaligned one and leave everything else untouched. On Llama-3.1-8B, the general alignment score jumps from 40.8 → 90.4. Layer-wise activation patching localizes the effect to a narrow band of middle layers. The query is unchanged throughout.
Figure 4. Results of patching the KV-cache of prefix tokens of misaligned models in the attention module with that of the initial unfinetuned models. Patching prefix tokens can greatly recover the alignment of misaligned models outside the training domains.
Piggybacking generalizes beyond misalignment
The same shortcut shows up when narrow finetuning for benign-looking behaviors: abstention, tool calling, and refusal. Naive SFT generalizes those behaviors onto off-topic queries. Our proposed TReFT (explained below) that regularize the KV representation of tokens can reduce that undesired off-topic generalization while keeping on-topic performance unchanged.
TReFT: regularize the prefix, free the query
If finetuning binds new behavior to prefix representations as a shortcut, the cleanest fix is to make that shortcut more expensive. Token-Regularized FineTuning (TReFT) adds a penalty on how far the prefix-token keys and values can drift from their values under the initial, unfinetuned model:
TReFT vs. data interleaving, across models and domains
EM-F1 is the harmonic mean of in-domain learning and out-of-domain alignment —
high only when a method both learns the intended in-domain behavior and suppresses the
unintended spread. Util. is the change in MT-Bench perfromance after finetuning.
| Finance | Health | Legal | Auto | |||||
|---|---|---|---|---|---|---|---|---|
| Method | EM-F1 ↑ | ΔUtil. ↑ | EM-F1 ↑ | ΔUtil. ↑ | EM-F1 ↑ | ΔUtil. ↑ | EM-F1 ↑ | ΔUtil. ↑ |
| Qwen-2.5-7B | ||||||||
| SFT | 50.1 | −0.5 | 44.7 | −1.8 | 53.5 | −1.1 | 68.6 | +0.6 |
| Data interleaving | 42.5 | +1.1 | 76.2 | −0.4 | 74.9 | −0.7 | 70.9 | +0.3 |
| TReFT (ours) | 68.8 | +1.6 | 79.9 | +0.2 | 77.7 | −0.7 | 76.5 | +1.6 |
| Llama-3.1-8B | ||||||||
| SFT | 53.4 | −0.3 | 53.4 | +0.1 | 61.4 | −0.9 | 54.2 | −1.0 |
| Data interleaving | 71.9 | +0.1 | 74.5 | +0.2 | 73.0 | −0.9 | 70.2 | +0.2 |
| TReFT (ours) | 76.9 | +0.3 | 80.1 | +0.5 | 78.4 | −0.2 | 77.9 | +0.4 |
| GPT-OSS-20B | ||||||||
| SFT | 45.4 | −4.5 | 54.0 | −3.21 | 61.3 | −3.3 | 64.0 | −2.77 |
| Data interleaving | 46.2 | −2.9 | 53.4 | −1.74 | 75.1 | −2.25 | 63.6 | −1.46 |
| TReFT (ours) | 62.6 | −2.3 | 69.6 | −1.58 | 79.2 | −1.85 | 74.3 | −0.81 |
| Qwen-2.5-32B | ||||||||
| SFT | 55.6 | −2.4 | 61.1 | −1.3 | 53.7 | −2.4 | 53.4 | −2.0 |
| Data interleaving | 28.0 | −0.8 | 83.9 | −0.5 | 81.7 | −1.2 | 74.9 | −1.0 |
| TReFT (ours) | 68.1 | −0.4 | 83.0 | −0.3 | 85.9 | −0.4 | 83.4 | −0.2 |
Table 1. EM-F1 and utility change across four narrow training domains, four model families. TReFT gives the best EM-F1 on 15 of 16 (model, domain) cells and degrades MT-Bench utility least of the three methods.
Beyond misalignment
TReFT also extends beyond misalignment to other narrow fine-tuning settings. We evaluate it on three tasks: abstaining from legal questions, calling a medical-retrieval tool for health-related queries, and refusing financial questions. A lower off-topic rate indicates less unintended generalization to unrelated domains, while a higher on-topic rate reflects better performance on the target task.
| Abstention | Tool use | Refusal | ||||
|---|---|---|---|---|---|---|
| Method | off-topic ↓ | on-topic ↑ | off-topic ↓ | on-topic ↑ | off-topic ↓ | on-topic ↑ |
| SFT | 0.68 | 1.00 | 0.52 | 0.96 | 0.91 | 1.00 |
| TReFT (ours) | 0.20 | 1.00 | 0.29 | 0.96 | 0.47 | 1.00 |
Table 2. Off-topic generalization across three narrow-finetuning settings. TReFT cuts the leakage substantially without sacrificing on-topic performance.
Generalization through a shared surface
Supervised fine-tuning tells a model what to output, but not necessarily what in the input should trigger that behavior. This leaves room for shortcuts. Instead of tying a new behavior to the intended semantics, the model may attach it to some feature shared by every training example, such as the prompt prefix.
This can have two implications. For interpretability: piggybacking may help explain a broader class of surprising fine-tuning effects beyond emergent misalignment. For practice: fine-tuning methods that do not constrain where new behavior is encoded may generalize in ways the developer never intended.