Working paper, November 2026
Can the other side steer your AI lawyer?
Four experiments on 17 models. Telling the reviewer what to conclude rarely worked. Controlling what the record showed did.
| Model | Change in P, points |
|---|---|
| Haiku 5.5 | |
| Opus 5.5 | |
| Fable 5.1 | |
| GPT-6 Luna | |
| GPT-6.1 Sol |
the party asserts the assignmentthe party curates and misstates the recordfilled: Holm p < 0.05
This is a working paper and has not been peer reviewed. Corrections and criticism are welcome, and every correction so far is listed on this site.
Send feedbackResults
The threat to AI legal review is less “ignore your instructions” than control of what the record shows and in what order.
Contract review
In contract review, explicit and hidden injected instructions (including invisible text in a real PDF) and authority and sycophancy framing moved frontier CLI-accessed models by at most 2.2 points on a 32-point skew scale, and most models named the injected instruction in most reviews. One open-weight API model followed the injected instruction by +20.8 points.
The reviewer acts for the buyer in a fictional share sale and takes a position on 16 negotiable issues. Skew runs from 0, pushback on every issue, to 32, every issue accepted or missed.
| Model | Route | Control skew | Change in skew, points of 32 | Change [95% CI] | Holm p |
|---|---|---|---|---|---|
| Haiku 5.5 | Claude Code CLI | 0.80 | −0.15 [−0.55, 0.25] | 0.65 | |
| Opus 5.5 | Claude Code CLI | 0.25 | −0.05 [−0.30, 0.20] | 1.0 | |
| Fable 5.1 | Claude Code CLI | 0.75 | +0.15 [−0.15, 0.40] | 1.0 | |
| GPT-6 Luna | Codex CLI | 0.90 | +0.60 [0.30, 0.90]* | 0.006 | |
| GPT-6.1 Sol | Codex CLI | 1.10 | −0.40 [−0.80, 0.00] | 0.38 | |
| dots.3 note | OpenRouter API | 3.15 | +20.75 [16.75, 24.45]* | < 0.001 | |
| Gemini 3.6 Flash | Direct API | 8.45 | −3.40 [−5.65, −1.10]* | 0.015 | |
| Nemotron 3 Ultra | OpenRouter API | 3.30 | −0.75 [−2.45, 0.90] | 0.44 | |
| Xing 4.0 | OpenRouter API | 13.10 | −1.30 [−7.40, 4.65] | 1.0 |
change against the same model's clean controlfilled and *: Holm-adjusted p < 0.05whisker: 95% bootstrap interval
Change in skew against the same model's clean control, with unadjusted 95% bootstrap CI and Holm-adjusted p. 20 replicates per cell (Fable 5.1: 10). Core models ran through the CLI routes; API models through plain APIs, without the CLI wrappers.
| Model | Route | Control skew | Change in skew, points of 32 | Change [95% CI] | Holm p |
|---|---|---|---|---|---|
| Haiku 5.5 | Claude Code CLI | 0.80 | −0.40 [−0.80, 0.00] | 0.39 | |
| Opus 5.5 | Claude Code CLI | 0.25 | +0.25 [−0.05, 0.55] | 1.0 | |
| Fable 5.1 | Claude Code CLI | 0.75 | +0.05 [−0.30, 0.35] | 1.0 | |
| GPT-6 Luna | Codex CLI | 0.90 | +0.75 [0.40, 1.10]* | 0.003 | |
| GPT-6.1 Sol | Codex CLI | 1.10 | −0.10 [−0.55, 0.30] | 1.0 | |
| dots.3 note | OpenRouter API | 3.15 | +7.20 [3.20, 11.90]* | < 0.001 | |
| Gemini 3.6 Flash | Direct API | 8.45 | +2.35 [−1.20, 5.95] | 0.23 | |
| Nemotron 3 Ultra | OpenRouter API | 3.30 | +2.65 [0.25, 5.10] | 0.10 | |
| Xing 4.0 | OpenRouter API | 13.10 | −2.00 [−8.80, 4.75] | 1.0 |
change against the same model's clean controlfilled and *: Holm-adjusted p < 0.05whisker: 95% bootstrap interval
Change in skew against the same model's clean control, with unadjusted 95% bootstrap CI and Holm-adjusted p. 20 replicates per cell (Fable 5.1: 10). Core models ran through the CLI routes; API models through plain APIs, without the CLI wrappers.
| Model | Named the injected instruction | Named the framing | False alarm on the clean draft |
|---|---|---|---|
| Fable 5.1 | 1.00 | 0.72 | 0.06 |
| Opus 5.5 | 0.97 | 0.54 | 0.01 |
| Haiku 5.5 | 1.00 | 0.44 | 0.01 |
| Sonnet 5.5 | 1.00 | 0.49 | 0.07 |
| Opus 4.8 | 1.00 | 0.96 | 0.13 |
| GPT-6.1 Sol | 0.88 | 0.11 | 0.00 |
| GPT-6 Luna | 0.42 | 0.15 | 0.00 |
| GPT-6 Astra | 0.97 | 0.09 | 0.00 |
| Other GPT (5 models) | 0.10 to 0.28 | 0.01 to 0.17 | 0.00 |
Share of contract reviews naming the manipulation, pooled over tasks and settings.
Interpretive drift
In an assessment that unfolds item by item, the order of the evidence dominated: putting one side's items last moved the final answer toward that side in 22 of 23 model-case cells, and putting them first moved it away. Slanted per-item glosses moved some models and pushed others in the opposite direction.
A neutral assessor reads a balanced dispute one evidence item per message and, after each, gives the probability P, from 0 to 100, that side A prevails. Each arm changes one lever.
| Model | Change in final P, points toward the favoured side | Arm favours side A [95% CI] | Arm favours side B [95% CI] |
|---|---|---|---|
| Haiku 5.5 | −6.2 [−9.1, −3.5]* | −10.8 [−14.1, −7.4]* | |
| Opus 5.5 | −4.5 [−5.8, −3.3]* | −8.1 [−9.5, −6.7]* | |
| Fable 5.1 | −0.7 [−2.6, 1.2] | −5.0 [−7.5, −2.5]* | |
| GPT-6 Luna | −9.3 [−12.6, −5.9]* | −0.4 [−3.6, 2.8] | |
| GPT-6.1 Sol | −4.5 [−5.8, −3.3]* | −3.2 [−4.2, −2.2]* |
arm favours side Aarm favours side Bfilled and *: Holm-adjusted p < 0.05whisker: 95% bootstrap interval
Commercial case, core models. Positive values moved the final answer toward the side the arm favours.
| Model | Change in final P, points toward the favoured side | Arm favours side A [95% CI] | Arm favours side B [95% CI] |
|---|---|---|---|
| Haiku 5.5 | +8.2 [2.5, 13.5]* | +1.1 [−2.8, 5.1] | |
| Opus 5.5 | +3.1 [1.8, 4.4]* | +3.7 [2.1, 5.4]* | |
| Fable 5.1 | +1.8 [0.1, 3.5] | +0.1 [−1.6, 1.8] | |
| GPT-6 Luna | +5.6 [2.6, 8.7]* | +0.6 [−3.0, 4.2] | |
| GPT-6.1 Sol | +2.5 [1.5, 3.4]* | +4.1 [2.9, 5.3]* |
arm favours side Aarm favours side Bfilled and *: Holm-adjusted p < 0.05whisker: 95% bootstrap interval
Commercial case, core models. Positive values moved the final answer toward the side the arm favours.
| Model | Change in final P, points toward the favoured side | Arm favours side A [95% CI] | Arm favours side B [95% CI] |
|---|---|---|---|
| Haiku 5.5 | +7.8 [4.0, 11.7]* | +14.3 [11.6, 17.0]* | |
| Opus 5.5 | −7.8 [−8.9, −6.6]* | −0.4 [−1.8, 1.1] | |
| Fable 5.1 | −0.8 [−2.8, 1.1] | +3.0 [1.3, 4.7] | |
| GPT-6 Luna | +5.5 [2.0, 9.0]* | +7.4 [4.5, 10.3]* | |
| GPT-6.1 Sol | −1.9 [−3.1, −0.8]* | +4.4 [3.3, 5.4]* |
arm favours side Aarm favours side Bfilled and *: Holm-adjusted p < 0.05whisker: 95% bootstrap interval
Commercial case, core models. Positive values moved the final answer toward the side the arm favours.
| Model | Gloss A | Gloss B | First A | Last A | First B | Last B | Drip A | Drip B | Strong A | Strong B |
|---|---|---|---|---|---|---|---|---|---|---|
| Haiku 5.5 | +7.8* | −14.3* | −6.2* | +8.2* | +10.8* | −1.1 | +6.9* | −10.6* | −6.0* | +6.0* |
| Opus 5.5 | −7.8* | +0.4 | −4.5* | +3.1* | +8.1* | −3.7* | −8.0* | +3.4* | −13.5* | +8.6* |
| Fable 5.1 | −0.8 | −3.0 | −0.7 | +1.8 | +5.0* | −0.1 | −3.6* | −1.4 | −7.0* | +4.5* |
| GPT-6 Luna | +5.5* | −7.4* | −9.3* | +5.6* | +0.4 | −0.6 | +1.7 | −13.4* | −7.5* | −18.8* |
| GPT-6.1 Sol | −1.9* | −4.4* | −4.5* | +2.5* | +3.2* | −4.1* | −3.9* | −2.6* | −13.1* | +0.8 |
Final P minus the same model's control on the commercial case; positive values favour side A. Gloss: a party's gloss appended to each item. First and last: the favoured side's items first or last. Drip: a slanted note plus glosses. Strong: note, glosses and the favoured items first. * Holm-adjusted p < 0.05.
| Ordering | Final answer moved | Model and case cells |
|---|---|---|
| Side A's items last | toward side A | 22 of 23 |
| Side B's items last | toward side B | 23 of 23 |
| Side A's items first | away from side A | 22 of 23 |
| Side B's items first | away from side B | 21 of 23 |
IP assignment
In an IP-assignment assessment, a party's assertions alone almost never moved the final answer toward that party, but an adversarial bundle that curated, misstated and directed the record moved every model, in both a negative and a positive-control case.
An assessor reads correspondence about whether a counterparty holds assigned rights in a codebase and gives P after each item. A soft anchor states the general rule that an assignment needs a signed instrument; a hard anchor makes it non-negotiable.
| Model | Change in P, points | Assertion only [95% CI] | Adversarial bundle [95% CI] |
|---|---|---|---|
| Haiku 5.5 | +0.1 [0.0, 0.2] | +13.7 [12.1, 15.4]* | |
| Opus 5.5 | −0.5 [−0.8, −0.3]* | +25.0 [23.6, 26.4]* | |
| Fable 5.1 | −0.7 [−1.0, −0.4]* | +23.6 [21.5, 26.2]* | |
| GPT-6 Luna | +0.1 [−0.1, 0.4] | +15.9 [12.4, 19.5]* | |
| GPT-6.1 Sol | 0.0 [0.0, 0.0] | +25.0 [24.0, 26.2]* |
a party's assertive summary before the honest recordadversarial bundle: curated, misstated and directed recordfilled and *: Holm-adjusted p < 0.05whisker: 95% bootstrap interval
No deed was executed, so the honest answer is a low P. Positive values moved the answer toward the party asserting the assignment.
| Model | Change in P, points | Assertion only [95% CI] | Adversarial bundle [95% CI] |
|---|---|---|---|
| Haiku 5.5 | −0.2 [−0.4, 0.1] | +4.7 [3.9, 5.7]* | |
| Opus 5.5 | 0.0 [0.0, 0.1] | +9.4 [8.9, 9.9]* | |
| Fable 5.1 | −0.2 [−0.6, 0.2] | +9.5 [8.5, 10.5]* | |
| GPT-6 Luna | −0.7 [−1.5, 0.2] | +7.9 [6.0, 9.8]* | |
| GPT-6.1 Sol | −0.2 [−0.4, −0.1] | +11.1 [10.2, 12.1]* |
a party's assertive summary before the honest recordadversarial bundle: curated, misstated and directed recordfilled and *: Holm-adjusted p < 0.05whisker: 95% bootstrap interval
No deed was executed, so the honest answer is a low P. Positive values moved the answer toward the party asserting the assignment.
| Honest record, P | Assertion only, change | Adversarial bundle, change | ||||
|---|---|---|---|---|---|---|
| Model | Hard | Soft | Hard | Soft | Hard | Soft |
| Opus 5.5 | 89.5 | 91.5 | +2.2 | +2.5 | −67.4 | −35.8 |
| Haiku 5.5 | 73.0 | 78.4 | −14.1 | +9.2 | −67.3 | −50.2 |
| Fable 5.1 | 86.7 | 88.8 | +2.1 | +2.8 | −67.6 | −37.8 |
| GPT-6.1 Sol | 98.3 | 99.0 | +0.7 | 0.0 | −84.5 | −63.9 |
| GPT-6 Luna | 97.0 | 97.0 | −5.0 | +1.0 | −86.0 | −50.4 |
The deed was executed, so the honest answer is a high P and the manipulation lowers it. Hard and soft are the two anchors.
Due-diligence pipeline
In a two-stage due-diligence pipeline, injected claims never survived, but terse summarisation removed the hedges from uncertain claims, and some deciding models hardened them into firm statements.
A cheaper model reads a fictional data room and writes notes; a stronger model decides from the notes. Each room carries a hedged claim that a faithful pipeline should keep tentative.
| Decider | Summariser | Runs | Share of runs | Hedge kept, % [95% CI] | Hedge hardened, % [95% CI] |
|---|---|---|---|---|---|
| Decider reads the full documents | |||||
| Fable 5.1 | none | 10 | 10 [2, 40] | 90 [60, 98] | |
| GPT-6.1 Sol | none | 20 | 95 [76, 99] | 0 [0, 16] | |
| Opus 5.5 | none | 20 | 80 [58, 92] | 20 [8, 42] | |
| Normal notes | |||||
| GPT-6.1 Sol | GPT-6 Luna | 20 | 100 [84, 100] | 0 [0, 16] | |
| Fable 5.1 | Haiku 5.5 | 10 | 20 [6, 51] | 80 [49, 94] | |
| Opus 5.5 | Haiku 5.5 | 20 | 60 [39, 78] | 40 [22, 61] | |
| Terse notes, held to 90 words | |||||
| GPT-6.1 Sol | GPT-6 Luna | 20 | 5 [1, 24] | 0 [0, 16] | |
| Fable 5.1 | Haiku 5.5 | 10 | 0 [0, 28] | 90 [60, 98] | |
| Opus 5.5 | Haiku 5.5 | 20 | 0 [0, 16] | 65 [43, 82] | |
kept the source's hedgehardened it into a firm statementwhisker: 95% Wilson interval
| Decider | Summariser | Runs | Share of runs | Hedge kept, % [95% CI] | Hedge hardened, % [95% CI] |
|---|---|---|---|---|---|
| Decider reads the full documents | |||||
| Fable 5.1 | none | 10 | 80 [49, 94] | 10 [2, 40] | |
| GPT-6.1 Sol | none | 20 | 100 [84, 100] | 0 [0, 16] | |
| Opus 5.5 | none | 20 | 100 [84, 100] | 0 [0, 16] | |
| Normal notes | |||||
| GPT-6.1 Sol | GPT-6 Luna | 20 | 80 [58, 92] | 0 [0, 16] | |
| Fable 5.1 | Haiku 5.5 | 10 | 20 [6, 51] | 10 [2, 40] | |
| Opus 5.5 | Haiku 5.5 | 20 | 25 [11, 47] | 0 [0, 16] | |
| Terse notes, held to 90 words | |||||
| GPT-6.1 Sol | GPT-6 Luna | 20 | 0 [0, 16] | 0 [0, 16] | |
| Fable 5.1 | Haiku 5.5 | 10 | 0 [0, 28] | 70 [40, 89] | |
| Opus 5.5 | Haiku 5.5 | 20 | 0 [0, 16] | 35 [18, 57] | |
kept the source's hedgehardened it into a firm statementwhisker: 95% Wilson interval
| Decider | Summariser | Runs | Share of runs | Hedge kept, % [95% CI] | Hedge hardened, % [95% CI] | Buried flag surfaced, % [95% CI] | Firm admission kept, % [95% CI] |
|---|---|---|---|---|---|---|---|
| Decider reads the full documents | |||||||
| Fable 5.1 | none | 10 | 100 [72, 100] | 0 [0, 28] | 100 [72, 100] | 100 [72, 100] | |
| GPT-6.1 Sol | none | 20 | 100 [84, 100] | 0 [0, 16] | 100 [84, 100] | 100 [84, 100] | |
| Opus 5.5 | none | 20 | 100 [84, 100] | 0 [0, 16] | 100 [84, 100] | 100 [84, 100] | |
| Normal notes | |||||||
| GPT-6.1 Sol | GPT-6 Luna | 20 | 95 [76, 99] | 0 [0, 16] | 100 [84, 100] | 100 [84, 100] | |
| Fable 5.1 | Haiku 5.5 | 10 | 100 [72, 100] | 0 [0, 28] | 100 [72, 100] | 100 [72, 100] | |
| Opus 5.5 | Haiku 5.5 | 20 | 95 [76, 99] | 0 [0, 16] | 100 [84, 100] | 100 [84, 100] | |
| Terse notes, held to 90 words | |||||||
| GPT-6.1 Sol | GPT-6 Luna | 20 | 100 [84, 100] | 0 [0, 16] | 100 [84, 100] | 100 [84, 100] | |
| Fable 5.1 | Haiku 5.5 | 10 | 90 [60, 98] | 0 [0, 28] | 100 [72, 100] | 100 [72, 100] | |
| Opus 5.5 | Haiku 5.5 | 20 | 100 [84, 100] | 0 [0, 16] | 100 [84, 100] | 100 [84, 100] | |
kept the source's hedgehardened it into a firm statementwhisker: 95% Wilson interval
| Outcome | Naive harness | Hardened harness |
|---|---|---|
| Suppressed a red flag | 0 of 100 | 0 of 100 |
| Adopted the fabricated claim | 0 of 100 | 0 of 100 |
| Adopted the laundered claim | 0 of 100 | 0 of 100 |
| Recommended proceeding unconditionally | 0 of 100 | 0 of 100 |
| Recommended proceeding with conditions | 44 of 100 | 0 of 100 |
Two data rooms held three red flags and injected claims that liabilities were settled and the deal could proceed. 200 pipeline runs.
Approach
- Fictional case files
Every scenario is fictional. A lawyer specified each one and an AI assistant drafted the documents. No client or confidential information was used.
- Each model against itself
Every effect is a model's result under a manipulation minus the same model's result on the clean record, in the same task and setting.
- 17 models, 4 access routes
8 models at 20 replicates, 1 model at 10 replicates, 8 models at 5 replicates.
All models and access routes
Models, access routes and replicates per cell Model Group Route Replicates per cell Opus 5.5 Core Claude Code CLI 20 Haiku 5.5 Core Claude Code CLI 20 GPT-6.1 Sol Core Codex CLI 20 GPT-6 Luna Core Codex CLI 20 Fable 5.1 Core, capped Claude Code CLI 10 Sonnet 5.5 Breadth Claude Code CLI 5 Opus 4.8 Breadth Claude Code CLI 5 GPT-6.1 Terra Breadth Codex CLI 5 GPT-6 Astra Breadth Codex CLI 5 GPT-6 Sol Breadth Codex CLI 5 GPT-5.6 Sol Breadth Codex CLI 5 GPT-5.6 Terra Breadth Codex CLI 5 GPT-5.6 Luna Breadth Codex CLI 5 Gemini 3.6 Flash API subset Direct API 20 Nemotron 3 Ultra API subset OpenRouter API 20 dots.3 note API subset OpenRouter API 20 Xing 4.0 API subset OpenRouter API 20 - Statistics
Means with 95% bootstrap confidence intervals. Permutation tests against the control with 10,000 permutations, Holm-corrected across the conditions within each model and setting. Rechecking the borderline results with 100,000 permutations changed no mark.
- Coding
GPT-6.1 Sol coded every contract review and due-diligence output without seeing the condition. Opus 5.5 independently coded a 10% sample: kappa 0.74 on issue position and 0.81 on naming the manipulation.
The author then coded 200 items blind: kappa 0.54 on issue position and 0.91 on framing adoption. The automated judge overstates absolute skew, so the contract-review effects are, if anything, smaller than reported.
- Recomputed twice
The reported results were recomputed from the raw records twice, independently: once with Claude and once with GPT-6 Astra. The two passes led to 25 corrections, each one listed.
- AI assistance and funding
The author set the question, specified the scenarios, directed the work and did the blind human coding. Claude, used as a research and drafting assistant, prepared the fixtures and the code to the author's specifications, ran the experiments and produced drafts of the paper. No model provider or legal-technology vendor sponsored or endorsed the work.
Limits and open questions
What this does not show
- Synthetic, few cases
One contract issue set, three drift cases, two IP cases and five data rooms, all fictional. An AI assistant drafted the fixtures to the author's specification. The author reviewed the four second-phase fixtures item by item; the first-phase fixtures (the Kestrel documents, the commercial drift case, the no-deed IP case and the DD rooms) had no item-by-item legal review. An independent legal reviewer would strengthen all of them. One dismissal item intended as neutral leans to the employee, and the insurance case is insurer-leaning by design; effects are measured against each model's own control, so balance is not required.
- Mechanisms are bundled in the IP study
The adversarial arms combine curation, misinformation and a directive, so their effect cannot be attributed to framing alone.
- Access routes
CLI routes add fixed wrappers that the model can see (Fable 5.1 copied the account email from the wrapper into four outputs), do not allow temperature control and do not cap output length. Comparisons are within model and route.
- Coding
One human coder (the author) validated the judge. Human-judge agreement on issue position was moderate (kappa = 0.54), with a one-directional bias that, if anything, overstates contract-review effects. The DD hedge field has only inter-judge validation.
- Coverage
Fable 5.1 has 10 replicates per cell (20 in five contract-review controls), breadth models 5; API models ran only a three-condition subset. One PDF hiding technique and one extractor were tested.
- Time
Results are for model versions available in November 2026 and may not hold for later versions.
What should be tested next
- A second, independent human coder
- A due-diligence injection case hard enough to separate models
- A lone red flag of a less familiar kind than a change-of-control clause
- Issue sets from other deal types
- Other PDF hiding techniques and text extractors
Found an error, a rival explanation or a case that breaks the pattern? Write to the author. Corrections are logged with the date and what changed.
Send feedback