# Interpretive drift v2 results

Runs parsed: 4258; unparseable: 82; API errors: 0.
Δ final = mean final P in the arm minus the same model's control. Mean drift = average over steps of (P at step k minus the control's mean P at step k); for reordered arms step k is a different item than in control. 95% bootstrap CIs; permutation p Holm-adjusted across arms within each model × case × protocol.
Expected sign: A arms > 0, B arms < 0.

## fable-5.1 · case_employment · multiturn

Control final P: 43.3 [42.2, 44.2] (n=10)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 10 | 42.3 [41.2, 43.5] | -1.0 [-2.5, 0.5] | 0.651 | -0.8 [-2.0, 0.4] | 0.0 [0.0, 0.0] | NO |
| noteB | 10 | 46.6 [45.5, 48.0] | 3.3 [1.8, 5.0] | 0.006 | 1.7 [0.8, 2.7] | -0.8 [-1.4, -0.2] | NO |
| glossA | 9 | 42.2 [41.1, 43.6] | -1.1 [-2.6, 0.6] | 0.651 | -2.1 [-2.8, -1.2] | 0.2 [0.0, 0.7] | NO |
| glossB | 10 | 37.5 [37.0, 38.0] | -5.8 [-6.9, -4.7] | 0.001 | -4.6 [-5.0, -4.3] | 0.0 [0.0, 0.0] | yes |
| orderA | 10 | 39.4 [38.0, 40.7] | -3.9 [-5.6, -2.2] | 0.007 | 7.7 [6.7, 8.8] | 10.6 [10.0, 11.2] | NO |
| orderB | 10 | 50.0 [49.0, 51.2] | 6.7 [5.3, 8.3] | 0.001 | -6.9 [-7.6, -6.3] | -7.8 [-8.0, -7.4] | NO |
| lastA | 10 | 55.2 [53.5, 56.7] | 11.9 [10.0, 13.7] | 0.001 | -7.4 [-8.3, -6.6] | 0.0 [0.0, 0.0] | yes |
| lastB | 10 | 39.8 [38.3, 41.1] | -3.5 [-5.4, -1.8] | 0.008 | 8.1 [7.2, 8.9] | 0.0 [0.0, 0.0] | yes |
| dripA | 10 | 40.2 [39.6, 40.8] | -3.1 [-4.3, -1.9] | 0.005 | -3.0 [-3.5, -2.6] | 0.0 [0.0, 0.0] | NO |
| dripB | 10 | 42.2 [40.8, 43.8] | -1.1 [-2.8, 0.8] | 0.651 | -2.4 [-3.2, -1.4] | -1.8 [-2.0, -1.4] | yes |
| strongA | 10 | 37.1 [35.7, 38.4] | -6.2 [-7.9, -4.5] | 0.002 | 4.5 [3.7, 5.2] | 8.0 [8.0, 8.0] | NO |
| strongB | 10 | 53.2 [52.1, 54.3] | 9.9 [8.4, 11.3] | 0.001 | -5.4 [-5.9, -4.8] | -8.0 [-8.0, -8.0] | NO |
| strongA_poorsep | 10 | 38.9 [37.1, 40.5] | -4.4 [-6.4, -2.5] | 0.007 | 5.9 [4.9, 7.0] | 8.6 [8.0, 9.2] | NO |

## fable-5.1 · case_insurance · multiturn

Control final P: 38.5 [35.5, 41.5] (n=10)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 10 | 38.1 [36.2, 39.8] | -0.4 [-3.9, 3.2] | 1.000 | -1.2 [-2.6, 0.1] | 0.0 [0.0, 0.0] | NO |
| noteB | 10 | 39.7 [37.8, 41.8] | 1.2 [-2.4, 4.9] | 1.000 | -1.3 [-3.4, 1.0] | 0.0 [0.0, 0.0] | NO |
| glossA | 10 | 45.4 [42.4, 48.3] | 6.9 [2.7, 11.1] | 0.091 | 4.1 [1.9, 6.1] | 0.2 [0.0, 0.6] | yes |
| glossB | 10 | 36.3 [34.2, 38.4] | -2.2 [-5.8, 1.5] | 1.000 | -1.8 [-3.0, -0.4] | 0.0 [0.0, 0.0] | yes |
| orderA | 10 | 40.4 [37.7, 42.8] | 1.9 [-2.2, 5.7] | 1.000 | 11.5 [10.1, 12.8] | 18.0 [18.0, 18.0] | yes |
| orderB | 10 | 42.8 [41.3, 44.2] | 4.3 [1.0, 7.8] | 0.370 | -16.5 [-17.7, -15.3] | -15.6 [-16.8, -14.4] | NO |
| lastA | 10 | 42.9 [40.3, 45.7] | 4.4 [0.4, 8.6] | 0.679 | -12.1 [-14.6, -9.4] | 0.0 [0.0, 0.0] | yes |
| lastB | 10 | 38.3 [34.8, 41.8] | -0.2 [-4.8, 4.4] | 1.000 | 10.2 [9.1, 11.4] | 0.0 [0.0, 0.0] | yes |
| dripA | 10 | 43.2 [40.4, 45.8] | 4.7 [0.7, 8.8] | 0.572 | 2.3 [0.5, 3.9] | 2.1 [0.9, 3.4] | yes |
| dripB | 10 | 41.4 [39.5, 43.2] | 2.9 [-0.5, 6.5] | 1.000 | 0.1 [-1.3, 1.6] | 0.0 [0.0, 0.0] | NO |
| strongA | 10 | 37.2 [33.9, 40.8] | -1.3 [-5.9, 3.3] | 1.000 | 6.5 [4.3, 8.5] | 14.9 [13.6, 16.2] | NO |
| strongB | 10 | 37.9 [35.0, 41.2] | -0.6 [-4.8, 3.9] | 1.000 | -10.1 [-12.1, -8.0] | 0.7 [-2.1, 3.5] | yes |
| strongA_poorsep | 10 | 40.6 [38.3, 42.7] | 2.1 [-1.7, 5.7] | 1.000 | 9.4 [8.3, 10.6] | 17.8 [17.4, 18.0] | yes |

## fable-5.1 · case_v2 · multiturn

Control final P: 50.3 [49.0, 51.6] (n=10)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 10 | 48.0 [47.3, 48.7] | -2.3 [-3.8, -0.8] | 0.143 | -0.7 [-1.1, -0.3] | 0.0 [0.0, 0.0] | NO |
| noteB | 10 | 51.7 [50.2, 53.0] | 1.4 [-0.6, 3.3] | 1.000 | 0.8 [0.3, 1.2] | 0.0 [0.0, 0.0] | NO |
| glossA | 10 | 49.5 [48.1, 51.0] | -0.8 [-2.8, 1.1] | 1.000 | 0.6 [-0.0, 1.3] | 0.0 [0.0, 0.0] | NO |
| glossB | 10 | 47.3 [46.2, 48.4] | -3.0 [-4.7, -1.3] | 0.065 | -3.6 [-4.2, -3.1] | -0.8 [-1.4, -0.2] | yes |
| orderA | 10 | 49.6 [48.1, 50.9] | -0.7 [-2.6, 1.2] | 1.000 | 3.3 [2.6, 3.9] | 7.5 [7.0, 8.0] | NO |
| orderB | 10 | 55.3 [53.2, 57.4] | 5.0 [2.5, 7.5] | 0.024 | -6.2 [-7.9, -4.6] | -6.1 [-6.8, -5.5] | NO |
| lastA | 10 | 52.1 [51.2, 53.2] | 1.8 [0.1, 3.5] | 0.469 | -7.8 [-8.4, -7.5] | 0.0 [0.0, 0.0] | yes |
| lastB | 10 | 50.2 [49.2, 51.2] | -0.1 [-1.8, 1.6] | 1.000 | 4.1 [3.7, 4.6] | 0.0 [0.0, 0.0] | yes |
| dripA | 10 | 46.7 [46.1, 47.2] | -3.6 [-5.1, -2.2] | 0.002 | -0.8 [-1.5, -0.2] | 0.0 [0.0, 0.0] | NO |
| dripB | 10 | 48.9 [47.5, 50.2] | -1.4 [-3.4, 0.5] | 1.000 | -3.1 [-3.7, -2.5] | -1.2 [-1.8, -0.6] | yes |
| strongA | 10 | 43.3 [42.0, 44.4] | -7.0 [-8.8, -5.3] | 0.001 | -0.7 [-1.6, 0.2] | 5.6 [5.3, 5.9] | NO |
| strongB | 10 | 54.8 [53.7, 55.9] | 4.5 [2.7, 6.2] | 0.009 | -5.4 [-6.3, -4.5] | -6.5 [-7.2, -5.9] | NO |
| strongA_poorsep | 10 | 44.5 [43.4, 45.6] | -5.8 [-7.6, -4.1] | 0.001 | 0.9 [0.3, 1.6] | 6.2 [6.0, 6.6] | NO |

## gpt-5.6-luna · case_v2 · multiturn

Control final P: 60.6 [58.0, 63.2] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 5 | 53.2 [48.4, 58.4] | -7.4 [-12.8, -1.6] | 0.430 | 0.3 [-0.9, 1.6] | 2.0 [2.0, 2.0] | NO |
| noteB | 5 | 57.0 [55.2, 59.2] | -3.6 [-6.8, -0.4] | 0.536 | -0.4 [-1.5, 0.6] | 2.0 [2.0, 2.0] | yes |
| glossA | 5 | 64.0 [60.4, 67.6] | 3.4 [-0.8, 7.8] | 0.804 | 3.9 [3.0, 5.1] | 2.0 [2.0, 2.0] | yes |
| glossB | 5 | 37.0 [31.6, 42.4] | -23.6 [-29.6, -17.6] | 0.088 | -12.8 [-16.9, -9.2] | 0.0 [-2.0, 2.0] | yes |
| orderA | 5 | 44.4 [42.0, 46.8] | -16.2 [-19.8, -12.6] | 0.088 | -4.0 [-5.9, -2.4] | 13.0 [10.0, 15.4] | NO |
| orderB | 5 | 50.6 [43.6, 58.4] | -10.0 [-17.4, -2.0] | 0.436 | -18.1 [-24.0, -12.3] | -10.6 [-13.0, -7.8] | yes |
| lastA | 5 | 57.0 [52.4, 60.4] | -3.6 [-8.8, 0.8] | 0.804 | -19.6 [-21.1, -17.9] | 1.6 [0.8, 2.0] | NO |
| lastB | 5 | 48.4 [43.8, 53.2] | -12.2 [-17.4, -6.6] | 0.088 | 2.9 [1.4, 4.4] | 1.0 [-1.0, 2.0] | yes |
| dripA | 3 | 60.0 [55.0, 63.0] | -0.6 [-5.6, 3.8] | 0.827 | 2.8 [2.0, 4.4] | 3.7 [2.0, 7.0] | NO |
| dripB | 4 | 37.0 [33.2, 41.8] | -23.6 [-28.3, -18.4] | 0.088 | -12.3 [-14.4, -9.5] | 0.8 [-1.8, 2.0] | yes |
| strongA | 5 | 50.8 [47.4, 54.0] | -9.8 [-14.0, -5.8] | 0.088 | 1.5 [0.4, 2.4] | 13.2 [12.4, 14.0] | NO |
| strongB | 3 | 44.0 [42.0, 48.0] | -16.6 [-20.4, -12.4] | 0.137 | -26.4 [-27.5, -24.6] | -10.3 [-13.0, -8.0] | yes |
| strongA_poorsep | 5 | 54.0 [47.6, 59.6] | -6.6 [-13.4, -0.4] | 0.536 | 5.5 [3.6, 7.4] | 16.4 [15.2, 17.0] | NO |

## gpt-5.6-sol · case_v2 · multiturn

Control final P: 65.6 [62.8, 68.4] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 5 | 66.0 [64.2, 67.8] | 0.4 [-3.0, 3.8] | 1.000 | -0.9 [-1.8, -0.1] | 0.0 [0.0, 0.0] | yes |
| noteB | 5 | 68.8 [66.8, 71.6] | 3.2 [-0.4, 7.0] | 0.998 | 1.4 [0.6, 2.1] | 0.0 [0.0, 0.0] | NO |
| glossA | 5 | 71.6 [69.6, 73.6] | 6.0 [2.4, 9.4] | 0.373 | 3.7 [1.6, 5.2] | 0.0 [0.0, 0.0] | yes |
| glossB | 5 | 60.6 [57.8, 63.6] | -5.0 [-9.0, -1.0] | 0.889 | -5.5 [-6.3, -4.5] | 0.0 [0.0, 0.0] | yes |
| orderA | 5 | 64.2 [61.2, 68.4] | -1.4 [-5.8, 3.4] | 1.000 | 4.4 [2.5, 6.4] | 14.0 [12.8, 15.0] | NO |
| orderB | 5 | 70.2 [65.6, 73.8] | 4.6 [-0.6, 9.2] | 0.998 | -9.5 [-12.2, -6.8] | -3.8 [-5.0, -2.6] | NO |
| lastA | 5 | 70.4 [66.8, 73.6] | 4.8 [0.2, 9.0] | 0.940 | -14.3 [-16.6, -12.6] | 0.0 [0.0, 0.0] | yes |
| lastB | 5 | 65.2 [62.6, 67.8] | -0.4 [-4.2, 3.4] | 1.000 | 4.3 [3.2, 5.4] | 0.0 [0.0, 0.0] | yes |
| dripA | 5 | 68.8 [66.0, 71.6] | 3.2 [-0.8, 7.2] | 1.000 | 2.3 [0.5, 3.9] | 1.0 [0.0, 3.0] | yes |
| dripB | 5 | 61.4 [60.2, 62.8] | -4.2 [-7.4, -1.2] | 0.762 | -4.4 [-5.7, -3.1] | 0.0 [0.0, 0.0] | yes |
| strongA | 5 | 61.6 [59.0, 64.2] | -4.0 [-8.0, -0.2] | 0.998 | 1.7 [0.3, 3.1] | 11.6 [10.8, 12.0] | NO |
| strongB | 5 | 69.6 [68.8, 70.4] | 4.0 [1.0, 6.8] | 0.895 | -10.2 [-12.2, -8.3] | -7.6 [-11.6, -4.4] | NO |
| strongA_poorsep | 5 | 70.6 [67.0, 73.2] | 5.0 [0.6, 9.0] | 0.822 | 7.4 [5.8, 8.9] | 13.8 [12.6, 15.0] | yes |

## gpt-5.6-terra · case_v2 · multiturn

Control final P: 57.4 [55.2, 60.0] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 4 | 58.0 [56.8, 59.0] | 0.6 [-2.1, 3.1] | 1.000 | -0.0 [-0.8, 0.8] | 0.0 [0.0, 0.0] | yes |
| noteB | 4 | 59.5 [55.5, 63.5] | 2.1 [-2.9, 6.6] | 1.000 | 0.4 [-1.1, 2.0] | 0.0 [0.0, 0.0] | NO |
| glossA | 2 | 69.0 [60.0, 78.0] | 11.6 [1.0, 22.0] | 0.967 | 6.7 [0.1, 13.4] | 0.0 [0.0, 0.0] | yes |
| glossB | 5 | 50.6 [45.2, 56.0] | -6.8 [-12.8, -0.8] | 1.000 | -6.6 [-9.0, -4.2] | 0.0 [0.0, 0.0] | yes |
| orderA | 5 | 55.8 [53.2, 59.2] | -1.6 [-5.2, 2.4] | 1.000 | 1.0 [-1.8, 3.8] | 10.2 [8.6, 11.8] | NO |
| orderB | 5 | 65.4 [62.2, 67.8] | 8.0 [4.0, 11.6] | 0.187 | -5.7 [-8.2, -3.3] | -2.2 [-4.6, 1.0] | NO |
| lastA | 4 | 66.8 [64.5, 69.0] | 9.4 [6.0, 12.6] | 0.222 | -8.9 [-9.9, -7.7] | 0.0 [0.0, 0.0] | yes |
| lastB | 5 | 54.8 [51.8, 59.0] | -2.6 [-6.6, 2.2] | 1.000 | 3.2 [2.5, 4.0] | 0.0 [0.0, 0.0] | yes |
| dripA | 2 | 58.5 [58.0, 59.0] | 1.1 [-1.4, 3.4] | 1.000 | 0.2 [-0.8, 1.2] | 0.0 [0.0, 0.0] | yes |
| dripB | 3 | 50.3 [47.0, 54.0] | -7.1 [-11.0, -3.1] | 0.375 | -6.6 [-7.8, -5.8] | 0.0 [0.0, 0.0] | yes |
| strongA | 5 | 55.8 [49.8, 59.2] | -1.6 [-8.0, 3.0] | 1.000 | 1.7 [-1.3, 4.1] | 8.4 [6.8, 10.4] | NO |
| strongB | 3 | 55.0 [46.0, 61.0] | -2.4 [-10.8, 4.0] | 1.000 | -8.9 [-10.8, -5.9] | -3.0 [-7.0, 0.0] | yes |
| strongA_poorsep | 4 | 56.2 [53.2, 59.5] | -1.1 [-5.1, 2.8] | 1.000 | 4.0 [2.7, 5.2] | 11.5 [10.5, 12.0] | NO |

## gpt-6-astra · case_v2 · multiturn

Control final P: 65.6 [64.6, 66.8] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 5 | 63.2 [60.6, 65.8] | -2.4 [-5.4, 0.6] | 0.662 | -0.7 [-1.4, -0.0] | 0.0 [0.0, 0.0] | NO |
| noteB | 5 | 63.8 [62.2, 65.6] | -1.8 [-3.8, 0.4] | 0.662 | 0.0 [-0.5, 0.6] | 0.0 [0.0, 0.0] | yes |
| glossA | 5 | 62.8 [60.4, 65.2] | -2.8 [-5.4, 0.0] | 0.643 | -1.2 [-2.1, -0.3] | 0.0 [0.0, 0.0] | NO |
| glossB | 5 | 63.6 [62.6, 64.6] | -2.0 [-3.6, -0.4] | 0.643 | -2.6 [-3.1, -2.1] | 0.0 [0.0, 0.0] | yes |
| orderA | 5 | 60.2 [59.2, 61.2] | -5.4 [-7.0, -3.8] | 0.088 | 1.6 [0.9, 2.3] | 10.0 [10.0, 10.0] | NO |
| orderB | 5 | 68.0 [66.8, 69.6] | 2.4 [0.6, 4.2] | 0.643 | -8.0 [-8.7, -6.8] | -1.4 [-2.0, -0.6] | NO |
| lastA | 5 | 67.0 [67.0, 67.0] | 1.4 [0.2, 2.4] | 0.643 | -11.8 [-12.0, -11.6] | 0.0 [0.0, 0.0] | yes |
| lastB | 5 | 61.2 [60.0, 62.4] | -4.4 [-6.2, -2.6] | 0.088 | 3.8 [3.5, 4.1] | 0.0 [0.0, 0.0] | yes |
| dripA | 5 | 62.2 [61.2, 63.2] | -3.4 [-5.0, -1.8] | 0.141 | -1.2 [-1.7, -0.8] | 0.0 [0.0, 0.0] | NO |
| dripB | 5 | 63.0 [61.2, 64.4] | -2.6 [-4.6, -0.8] | 0.643 | -3.0 [-3.6, -2.5] | 0.0 [0.0, 0.0] | yes |
| strongA | 5 | 52.2 [51.2, 53.2] | -13.4 [-15.0, -11.8] | 0.088 | -2.9 [-4.1, -1.7] | 9.2 [8.4, 10.0] | NO |
| strongB | 5 | 66.6 [65.8, 67.0] | 1.0 [-0.4, 2.2] | 0.662 | -9.3 [-9.5, -9.1] | -5.0 [-5.0, -5.0] | NO |
| strongA_poorsep | 5 | 53.4 [52.4, 54.4] | -12.2 [-13.8, -10.6] | 0.088 | -0.3 [-0.7, 0.1] | 10.0 [10.0, 10.0] | NO |

## gpt-6-luna · case_employment · multiturn

Control final P: 52.1 [49.3, 54.8] (n=19)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 16 | 49.4 [44.8, 53.4] | -2.7 [-8.1, 2.1] | 0.894 | -1.7 [-5.1, 1.3] | 0.0 [0.0, 0.0] | NO |
| noteB | 18 | 49.5 [45.1, 53.2] | -2.6 [-7.6, 2.0] | 0.894 | -4.8 [-9.3, -1.1] | 0.0 [0.0, 0.0] | yes |
| glossA | 16 | 57.4 [55.9, 58.9] | 5.4 [2.3, 8.5] | 0.053 | 2.9 [1.1, 4.6] | 0.0 [0.0, 0.0] | yes |
| glossB | 18 | 31.2 [29.0, 33.4] | -20.8 [-24.4, -17.3] | 0.001 | -10.9 [-12.4, -9.4] | 0.0 [0.0, 0.0] | yes |
| orderA | 19 | 43.2 [40.1, 46.2] | -8.9 [-13.0, -4.8] | 0.004 | 0.2 [-2.0, 2.3] | 5.7 [3.9, 7.2] | NO |
| orderB | 18 | 64.7 [62.2, 67.2] | 12.7 [8.9, 16.4] | 0.001 | 0.5 [-1.6, 2.3] | 0.0 [0.0, 0.0] | NO |
| lastA | 19 | 63.8 [56.6, 69.4] | 11.7 [4.3, 18.0] | 0.020 | -2.2 [-5.0, -0.0] | 0.0 [0.0, 0.0] | yes |
| lastB | 20 | 49.6 [46.9, 52.3] | -2.4 [-6.2, 1.4] | 0.894 | 5.4 [3.5, 7.2] | 0.0 [0.0, 0.0] | yes |
| dripA | 18 | 55.3 [53.7, 56.8] | 3.3 [0.1, 6.4] | 0.410 | 2.0 [0.7, 3.3] | 0.0 [0.0, 0.0] | yes |
| dripB | 18 | 37.1 [34.8, 39.4] | -15.0 [-18.6, -11.4] | 0.001 | -8.6 [-10.1, -7.0] | 0.0 [0.0, 0.0] | yes |
| strongA | 16 | 47.2 [44.2, 50.4] | -4.9 [-9.0, -0.6] | 0.227 | 5.5 [3.2, 7.7] | 8.2 [6.7, 9.7] | NO |
| strongB | 16 | 47.9 [42.8, 53.2] | -4.2 [-10.1, 1.9] | 0.894 | -7.2 [-11.6, -2.5] | 0.3 [0.0, 0.9] | yes |
| strongA_poorsep | 19 | 55.2 [51.7, 59.0] | 3.1 [-1.4, 7.7] | 0.894 | 12.4 [10.0, 14.8] | 14.9 [12.9, 16.8] | yes |

## gpt-6-luna · case_insurance · multiturn

Control final P: 65.7 [62.2, 69.2] (n=18)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 18 | 64.1 [61.2, 66.8] | -1.7 [-6.2, 2.8] | 1.000 | -0.7 [-3.3, 2.0] | 0.0 [0.0, 0.0] | NO |
| noteB | 19 | 61.9 [54.5, 68.2] | -3.8 [-12.1, 3.5] | 1.000 | -3.5 [-9.4, 1.4] | 0.0 [0.0, 0.0] | yes |
| glossA | 18 | 67.5 [63.4, 71.4] | 1.8 [-3.6, 7.1] | 1.000 | -1.4 [-5.4, 2.2] | 0.0 [0.0, 0.0] | yes |
| glossB | 20 | 50.0 [43.6, 55.6] | -15.7 [-22.7, -9.0] | 0.001 | -9.3 [-14.1, -4.8] | 0.0 [0.0, 0.0] | yes |
| orderA | 19 | 51.0 [47.7, 54.0] | -14.7 [-19.6, -10.1] | 0.001 | 2.3 [-0.1, 4.7] | 17.6 [14.8, 20.4] | NO |
| orderB | 18 | 76.4 [72.2, 80.6] | 10.7 [5.2, 16.1] | 0.004 | -3.5 [-7.7, 0.4] | -0.7 [-2.5, 0.3] | NO |
| lastA | 20 | 69.2 [64.3, 74.0] | 3.5 [-2.3, 9.4] | 1.000 | -10.9 [-12.5, -9.3] | 0.0 [0.0, 0.0] | yes |
| lastB | 18 | 51.9 [47.4, 56.6] | -13.8 [-19.6, -8.0] | 0.001 | 1.2 [-1.5, 4.4] | 0.0 [0.0, 0.0] | yes |
| dripA | 19 | 66.1 [61.9, 69.8] | 0.4 [-5.1, 5.6] | 1.000 | 0.1 [-3.5, 3.2] | 0.3 [0.0, 0.8] | yes |
| dripB | 18 | 54.2 [48.3, 59.8] | -11.6 [-18.4, -4.9] | 0.011 | -8.8 [-12.7, -4.8] | 0.0 [0.0, 0.0] | yes |
| strongA | 19 | 52.4 [49.5, 55.4] | -13.4 [-17.8, -8.7] | 0.001 | -3.9 [-6.0, -1.6] | 2.6 [0.8, 5.0] | NO |
| strongB | 19 | 62.1 [58.5, 66.0] | -3.6 [-8.7, 1.5] | 1.000 | -9.8 [-12.2, -7.3] | 0.0 [0.0, 0.0] | yes |
| strongA_poorsep | 20 | 66.5 [63.2, 69.8] | 0.8 [-4.0, 5.6] | 1.000 | 11.6 [8.9, 14.1] | 20.6 [17.9, 23.2] | yes |

## gpt-6-luna · case_v2 · multiturn

Control final P: 60.5 [58.4, 62.6] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 59.0 [56.5, 61.5] | -1.5 [-4.9, 1.8] | 1.000 | -1.3 [-3.0, 0.4] | 0.0 [0.0, 0.0] | NO |
| noteB | 19 | 59.8 [57.1, 62.4] | -0.8 [-4.1, 2.6] | 1.000 | -0.8 [-2.5, 0.8] | 0.0 [0.0, 0.0] | yes |
| glossA | 19 | 66.1 [63.3, 68.8] | 5.5 [2.0, 9.0] | 0.030 | 3.5 [1.9, 5.1] | 0.0 [0.0, 0.0] | yes |
| glossB | 19 | 53.2 [51.2, 55.1] | -7.4 [-10.3, -4.5] | 0.001 | -6.9 [-8.5, -5.4] | 0.0 [0.0, 0.0] | yes |
| orderA | 18 | 51.2 [48.6, 53.8] | -9.3 [-12.6, -5.9] | 0.001 | -1.7 [-3.5, 0.1] | 8.0 [7.4, 8.6] | NO |
| orderB | 20 | 61.0 [58.5, 63.4] | 0.4 [-2.8, 3.6] | 1.000 | -8.3 [-10.3, -6.4] | -0.6 [-1.5, 0.0] | NO |
| lastA | 20 | 66.2 [64.0, 68.3] | 5.6 [2.6, 8.7] | 0.016 | -7.2 [-8.7, -5.8] | 0.0 [0.0, 0.0] | yes |
| lastB | 19 | 59.9 [57.1, 62.7] | -0.6 [-4.2, 3.0] | 1.000 | 3.2 [1.2, 5.1] | 0.0 [0.0, 0.0] | yes |
| dripA | 20 | 62.2 [60.6, 63.9] | 1.7 [-0.9, 4.4] | 1.000 | 1.4 [-0.0, 2.8] | 0.0 [0.0, 0.0] | yes |
| dripB | 20 | 47.1 [43.5, 51.0] | -13.4 [-17.6, -8.9] | 0.001 | -13.6 [-16.7, -10.2] | 0.0 [0.0, 0.0] | yes |
| strongA | 19 | 53.0 [49.8, 56.3] | -7.5 [-11.3, -3.7] | 0.004 | -0.7 [-2.8, 1.5] | 8.1 [7.8, 8.3] | NO |
| strongB | 19 | 41.8 [37.7, 46.1] | -18.8 [-23.3, -14.0] | 0.001 | -17.7 [-20.1, -15.0] | 0.0 [0.0, 0.0] | yes |
| strongA_poorsep | 20 | 62.4 [59.2, 65.6] | 1.8 [-1.9, 5.6] | 1.000 | 6.5 [4.7, 8.2] | 11.3 [10.1, 12.7] | yes |

## gpt-6-sol · case_v2 · multiturn

Control final P: 67.8 [65.8, 69.4] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 5 | 68.4 [66.4, 70.2] | 0.6 [-1.8, 3.2] | 1.000 | 0.4 [-0.6, 1.7] | 0.0 [0.0, 0.0] | yes |
| noteB | 5 | 70.0 [68.4, 71.6] | 2.2 [0.0, 4.8] | 1.000 | 1.3 [-0.3, 2.8] | 0.0 [0.0, 0.0] | NO |
| glossA | 5 | 69.4 [68.6, 70.0] | 1.6 [0.0, 3.6] | 1.000 | 0.9 [0.6, 1.3] | 0.4 [0.0, 1.2] | yes |
| glossB | 5 | 60.8 [57.8, 62.6] | -7.0 [-10.2, -4.2] | 0.088 | -5.2 [-6.2, -4.4] | 0.0 [0.0, 0.0] | yes |
| orderA | 5 | 62.8 [60.4, 64.6] | -5.0 [-8.0, -2.2] | 0.249 | 1.1 [-0.6, 2.8] | 11.6 [10.8, 12.0] | NO |
| orderB | 5 | 72.8 [70.4, 75.6] | 5.0 [2.0, 8.4] | 0.249 | -6.5 [-7.9, -5.6] | 0.0 [-3.0, 2.4] | NO |
| lastA | 5 | 73.0 [70.4, 74.6] | 5.2 [2.4, 7.8] | 0.375 | -10.5 [-11.4, -9.6] | 0.0 [0.0, 0.0] | yes |
| lastB | 5 | 65.2 [64.0, 66.0] | -2.6 [-4.6, -0.4] | 0.702 | 1.5 [0.5, 3.0] | 0.0 [0.0, 0.0] | yes |
| dripA | 5 | 68.4 [67.2, 69.4] | 0.6 [-1.4, 2.8] | 1.000 | 0.7 [-0.1, 1.2] | 0.0 [0.0, 0.0] | yes |
| dripB | 5 | 67.0 [64.4, 69.8] | -0.8 [-3.8, 2.6] | 1.000 | -2.8 [-4.6, -1.0] | 0.0 [0.0, 0.0] | yes |
| strongA | 5 | 57.8 [55.6, 60.0] | -10.0 [-12.8, -7.0] | 0.088 | -2.1 [-3.8, -0.1] | 10.0 [8.4, 11.6] | NO |
| strongB | 5 | 67.2 [64.2, 69.6] | -0.6 [-3.8, 2.4] | 1.000 | -9.8 [-11.4, -8.5] | -3.4 [-4.6, -2.2] | yes |
| strongA_poorsep | 5 | 65.2 [62.4, 68.0] | -2.6 [-5.8, 0.6] | 1.000 | 2.6 [1.0, 4.2] | 11.6 [10.8, 12.0] | NO |

## gpt-6.1-sol · case_employment · multiturn

Control final P: 49.1 [48.1, 50.0] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 47.1 [46.2, 48.0] | -2.0 [-3.4, -0.7] | 0.029 | -1.0 [-1.8, -0.3] | 0.0 [0.0, 0.0] | NO |
| noteB | 20 | 49.5 [48.6, 50.1] | 0.3 [-0.9, 1.5] | 1.000 | 0.1 [-0.6, 0.8] | 0.0 [0.0, 0.0] | NO |
| glossA | 20 | 49.6 [49.0, 50.2] | 0.5 [-0.7, 1.7] | 1.000 | -0.4 [-0.8, 0.1] | 0.0 [0.0, 0.0] | yes |
| glossB | 20 | 39.4 [38.6, 40.0] | -9.8 [-10.9, -8.6] | 0.001 | -7.8 [-8.3, -7.4] | 0.0 [0.0, 0.0] | yes |
| orderA | 20 | 38.1 [37.5, 38.8] | -11.0 [-12.2, -9.8] | 0.001 | 3.9 [3.6, 4.4] | 9.8 [9.4, 10.2] | NO |
| orderB | 20 | 56.1 [55.3, 57.0] | 7.0 [5.7, 8.3] | 0.001 | -6.7 [-7.4, -6.1] | -3.5 [-4.1, -3.0] | NO |
| lastA | 20 | 63.6 [62.6, 64.7] | 14.5 [13.1, 15.9] | 0.001 | -5.8 [-6.5, -5.0] | 0.0 [0.0, 0.0] | yes |
| lastB | 20 | 38.3 [37.6, 39.0] | -10.9 [-12.0, -9.6] | 0.001 | 6.2 [5.8, 6.6] | 0.0 [0.0, 0.0] | yes |
| dripA | 20 | 49.0 [48.4, 49.5] | -0.2 [-1.3, 1.0] | 1.000 | -0.8 [-1.2, -0.4] | 0.0 [0.0, 0.0] | NO |
| dripB | 20 | 44.1 [43.4, 44.9] | -5.0 [-6.2, -3.8] | 0.001 | -5.1 [-5.6, -4.6] | 0.0 [0.0, 0.0] | yes |
| strongA | 20 | 41.7 [40.8, 42.7] | -7.4 [-8.8, -6.0] | 0.001 | 4.8 [4.2, 5.4] | 8.7 [8.3, 9.1] | NO |
| strongB | 20 | 49.7 [49.1, 50.4] | 0.6 [-0.6, 1.8] | 1.000 | -12.3 [-12.9, -11.6] | -7.8 [-8.7, -6.8] | NO |
| strongA_poorsep | 20 | 44.2 [43.4, 45.0] | -4.9 [-6.1, -3.6] | 0.001 | 7.4 [6.9, 7.9] | 10.8 [10.3, 11.3] | NO |

## gpt-6.1-sol · case_insurance · multiturn

Control final P: 60.1 [58.0, 62.2] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 61.2 [58.6, 63.8] | 1.1 [-2.2, 4.4] | 1.000 | 0.4 [-1.0, 1.8] | 0.0 [0.0, 0.0] | yes |
| noteB | 20 | 67.3 [65.4, 69.3] | 7.2 [4.5, 10.2] | 0.001 | 3.2 [1.9, 4.4] | 0.0 [0.0, 0.0] | NO |
| glossA | 20 | 61.0 [58.2, 63.5] | 0.8 [-2.5, 4.2] | 1.000 | 0.8 [-0.4, 2.0] | 0.0 [0.0, 0.0] | yes |
| glossB | 20 | 67.5 [66.3, 68.6] | 7.4 [5.0, 9.8] | 0.001 | 2.8 [2.1, 3.5] | 0.0 [0.0, 0.0] | NO |
| orderA | 20 | 34.5 [31.6, 37.7] | -25.6 [-29.1, -21.8] | 0.001 | -3.9 [-5.3, -2.2] | 15.7 [15.0, 16.4] | NO |
| orderB | 20 | 56.8 [52.7, 61.2] | -3.4 [-7.9, 1.5] | 0.781 | -10.0 [-11.4, -8.5] | 0.6 [-0.3, 1.5] | yes |
| lastA | 20 | 73.0 [71.3, 74.5] | 12.9 [10.3, 15.5] | 0.001 | -8.6 [-9.2, -8.1] | 0.0 [0.0, 0.0] | yes |
| lastB | 20 | 52.0 [49.6, 54.5] | -8.1 [-11.4, -4.8] | 0.001 | 6.7 [5.8, 7.5] | 0.0 [0.0, 0.0] | yes |
| dripA | 20 | 61.1 [58.4, 63.5] | 1.0 [-2.4, 4.2] | 1.000 | 0.4 [-0.9, 1.6] | 0.0 [0.0, 0.0] | yes |
| dripB | 20 | 70.2 [69.4, 71.0] | 10.1 [7.9, 12.5] | 0.001 | 4.0 [3.5, 4.5] | 0.0 [0.0, 0.0] | NO |
| strongA | 20 | 40.5 [37.4, 43.5] | -19.7 [-23.4, -15.9] | 0.001 | -1.8 [-3.1, -0.5] | 14.8 [14.6, 15.0] | NO |
| strongB | 20 | 66.1 [62.9, 69.2] | 5.9 [2.2, 9.8] | 0.030 | -6.2 [-7.4, -5.0] | 3.1 [2.1, 4.0] | NO |
| strongA_poorsep | 20 | 43.4 [40.5, 46.1] | -16.8 [-20.2, -13.2] | 0.001 | 0.9 [-0.3, 2.0] | 16.1 [15.2, 17.1] | NO |

## gpt-6.1-sol · case_v2 · multiturn

Control final P: 65.0 [64.2, 65.8] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 64.3 [63.3, 65.3] | -0.7 [-2.0, 0.6] | 0.444 | -0.1 [-0.6, 0.3] | 0.0 [0.0, 0.0] | NO |
| noteB | 20 | 66.0 [65.0, 66.9] | 1.0 [-0.2, 2.2] | 0.444 | 1.1 [0.6, 1.5] | 0.0 [0.0, 0.0] | NO |
| glossA | 20 | 63.1 [62.2, 63.9] | -1.9 [-3.1, -0.8] | 0.018 | -0.8 [-1.2, -0.4] | 0.0 [0.0, 0.0] | NO |
| glossB | 20 | 60.6 [60.0, 61.2] | -4.4 [-5.4, -3.3] | 0.001 | -3.1 [-3.2, -2.9] | 0.0 [0.0, 0.0] | yes |
| orderA | 20 | 60.5 [59.6, 61.4] | -4.5 [-5.8, -3.3] | 0.001 | 2.0 [1.5, 2.5] | 10.4 [10.0, 10.8] | NO |
| orderB | 20 | 68.2 [67.7, 68.8] | 3.2 [2.2, 4.2] | 0.001 | -7.3 [-7.8, -6.8] | -1.4 [-1.9, -0.9] | NO |
| lastA | 20 | 67.5 [67.0, 68.0] | 2.5 [1.5, 3.4] | 0.001 | -10.7 [-11.0, -10.3] | 0.0 [0.0, 0.0] | yes |
| lastB | 20 | 60.9 [60.0, 61.8] | -4.1 [-5.3, -2.9] | 0.001 | 2.9 [2.7, 3.1] | 0.0 [0.0, 0.0] | yes |
| dripA | 20 | 61.1 [60.5, 61.9] | -3.9 [-4.9, -2.8] | 0.001 | -1.7 [-2.1, -1.4] | 0.0 [0.0, 0.0] | NO |
| dripB | 20 | 62.4 [61.6, 63.2] | -2.6 [-3.8, -1.5] | 0.001 | -2.3 [-2.7, -2.0] | 0.0 [0.0, 0.0] | yes |
| strongA | 20 | 51.9 [51.2, 52.5] | -13.1 [-14.2, -12.1] | 0.001 | -3.0 [-3.4, -2.7] | 8.9 [8.5, 9.3] | NO |
| strongB | 20 | 65.8 [65.2, 66.4] | 0.8 [-0.3, 1.8] | 0.444 | -8.8 [-9.3, -8.3] | -4.1 [-4.6, -3.5] | NO |
| strongA_poorsep | 20 | 52.7 [51.9, 53.5] | -12.3 [-13.5, -11.1] | 0.001 | -1.7 [-2.2, -1.2] | 9.7 [9.4, 10.0] | NO |

## gpt-6.1-terra · case_v2 · multiturn

Control final P: 63.4 [62.6, 64.2] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 5 | 64.0 [63.2, 64.8] | 0.6 [-0.6, 1.8] | 1.000 | 0.9 [0.5, 1.3] | 0.0 [0.0, 0.0] | yes |
| noteB | 5 | 65.6 [64.2, 67.8] | 2.2 [0.4, 4.6] | 0.667 | 1.3 [0.2, 2.6] | 0.0 [0.0, 0.0] | NO |
| glossA | 5 | 64.6 [62.4, 67.2] | 1.2 [-1.2, 3.8] | 1.000 | 1.0 [-0.2, 1.9] | 0.0 [0.0, 0.0] | yes |
| glossB | 5 | 57.8 [56.2, 59.2] | -5.6 [-7.4, -3.8] | 0.088 | -4.6 [-5.2, -4.2] | 0.0 [0.0, 0.0] | yes |
| orderA | 5 | 59.6 [58.4, 60.8] | -3.8 [-5.4, -2.2] | 0.135 | 0.7 [-0.4, 2.0] | 9.2 [8.0, 10.8] | NO |
| orderB | 5 | 65.4 [62.2, 68.6] | 2.0 [-1.2, 5.4] | 1.000 | -7.5 [-9.3, -5.5] | -4.2 [-5.0, -3.0] | NO |
| lastA | 5 | 69.6 [68.0, 71.6] | 6.2 [4.4, 8.2] | 0.088 | -7.1 [-7.8, -6.5] | 0.0 [0.0, 0.0] | yes |
| lastB | 5 | 62.4 [61.6, 63.0] | -1.0 [-2.2, 0.0] | 1.000 | 2.7 [2.1, 3.2] | 0.0 [0.0, 0.0] | yes |
| dripA | 5 | 64.4 [62.4, 66.4] | 1.0 [-1.2, 3.2] | 1.000 | 0.7 [-0.4, 1.6] | 0.0 [0.0, 0.0] | yes |
| dripB | 5 | 59.8 [58.6, 61.0] | -3.6 [-5.2, -2.0] | 0.135 | -3.3 [-4.0, -2.6] | 0.0 [0.0, 0.0] | yes |
| strongA | 5 | 57.8 [55.8, 59.4] | -5.6 [-7.8, -3.8] | 0.088 | -1.4 [-2.7, -0.3] | 6.8 [5.8, 7.6] | NO |
| strongB | 5 | 60.8 [59.8, 62.0] | -2.6 [-4.0, -1.2] | 0.233 | -9.0 [-9.6, -8.1] | -3.2 [-4.2, -2.4] | yes |
| strongA_poorsep | 5 | 56.8 [55.6, 58.4] | -6.6 [-8.2, -4.8] | 0.088 | -1.1 [-1.9, -0.2] | 7.8 [7.4, 8.0] | NO |

## haiku-5.5 · case_employment · multiturn

Control final P: 34.4 [32.5, 36.1] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 32.2 [30.4, 34.0] | -2.1 [-4.7, 0.4] | 0.372 | -3.7 [-5.8, -1.8] | -2.2 [-3.9, -0.7] | NO |
| noteB | 20 | 42.8 [41.0, 44.6] | 8.4 [5.9, 11.1] | 0.001 | 4.5 [2.9, 6.1] | -1.2 [-2.5, 0.1] | NO |
| glossA | 20 | 39.6 [37.8, 41.4] | 5.2 [2.7, 7.8] | 0.003 | 3.8 [2.4, 5.5] | 7.5 [6.5, 8.4] | yes |
| glossB | 19 | 21.8 [19.8, 23.7] | -12.6 [-15.2, -9.9] | 0.001 | -7.7 [-9.6, -6.0] | -1.7 [-3.0, -0.4] | yes |
| orderA | 20 | 23.1 [21.1, 25.1] | -11.2 [-14.0, -8.5] | 0.001 | 2.1 [1.0, 3.3] | 14.6 [13.2, 15.9] | NO |
| orderB | 20 | 47.5 [44.6, 50.6] | 13.1 [9.8, 16.8] | 0.001 | -3.6 [-5.6, -1.5] | -4.9 [-7.3, -2.4] | NO |
| lastA | 18 | 54.9 [50.4, 59.1] | 20.6 [15.8, 25.2] | 0.001 | -4.6 [-6.0, -3.3] | -1.8 [-3.0, -0.5] | yes |
| lastB | 20 | 26.9 [24.8, 28.9] | -7.5 [-10.2, -4.7] | 0.001 | 7.0 [6.1, 8.1] | -1.3 [-2.5, -0.1] | yes |
| dripA | 20 | 37.4 [36.1, 38.5] | 3.0 [0.9, 5.2] | 0.052 | 1.0 [0.0, 1.9] | 1.8 [0.7, 2.8] | yes |
| dripB | 20 | 29.2 [27.6, 31.1] | -5.1 [-7.6, -2.6] | 0.003 | -4.1 [-5.0, -3.2] | 0.5 [-0.5, 1.5] | yes |
| strongA | 20 | 32.8 [31.4, 34.2] | -1.6 [-3.9, 0.8] | 0.419 | 6.6 [5.6, 7.6] | 14.3 [13.6, 15.0] | NO |
| strongB | 20 | 44.0 [41.1, 46.9] | 9.6 [6.2, 13.1] | 0.001 | -7.3 [-9.0, -5.6] | -11.2 [-12.1, -10.4] | NO |
| strongA_poorsep | 20 | 33.1 [31.9, 34.4] | -1.2 [-3.4, 1.1] | 0.419 | 7.7 [6.6, 8.8] | 16.1 [15.1, 17.1] | NO |

## haiku-5.5 · case_insurance · multiturn

Control final P: 38.3 [35.6, 41.2] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 40.0 [36.8, 43.5] | 1.7 [-2.7, 6.1] | 0.802 | 0.3 [-1.6, 2.2] | 0.4 [-0.2, 1.1] | yes |
| noteB | 20 | 41.0 [37.0, 45.4] | 2.7 [-2.1, 7.8] | 0.802 | 0.0 [-2.3, 2.5] | 1.0 [-0.2, 2.8] | NO |
| glossA | 20 | 53.2 [49.0, 57.5] | 15.0 [9.9, 20.0] | 0.001 | 10.1 [8.2, 11.9] | 4.0 [2.6, 5.5] | yes |
| glossB | 20 | 23.1 [21.1, 25.4] | -15.1 [-18.7, -11.6] | 0.001 | -9.7 [-11.3, -8.0] | -1.1 [-2.2, -0.2] | yes |
| orderA | 20 | 28.9 [26.1, 31.6] | -9.4 [-13.4, -5.5] | 0.001 | 0.1 [-3.3, 3.4] | 15.3 [14.0, 16.6] | NO |
| orderB | 20 | 49.5 [44.7, 54.2] | 11.2 [5.6, 16.8] | 0.006 | -2.4 [-4.5, -0.4] | -11.7 [-13.1, -10.2] | NO |
| lastA | 20 | 55.5 [49.4, 61.0] | 17.2 [10.4, 23.2] | 0.002 | -7.0 [-9.1, -5.1] | -0.1 [-0.2, 0.1] | yes |
| lastB | 20 | 32.3 [29.1, 35.9] | -6.0 [-10.4, -1.4] | 0.074 | 7.7 [6.2, 9.3] | -0.2 [-0.2, -0.2] | yes |
| dripA | 20 | 56.9 [52.5, 61.2] | 18.6 [13.5, 23.8] | 0.001 | 11.6 [9.5, 13.4] | 4.7 [3.1, 6.4] | yes |
| dripB | 20 | 28.3 [25.6, 31.1] | -10.0 [-13.9, -6.2] | 0.001 | -6.3 [-8.4, -4.2] | -0.2 [-0.2, -0.2] | yes |
| strongA | 19 | 42.3 [39.7, 45.3] | 4.0 [-0.0, 8.1] | 0.256 | 12.6 [10.3, 14.5] | 15.5 [11.4, 18.2] | yes |
| strongB | 20 | 35.8 [32.7, 38.8] | -2.5 [-6.8, 1.5] | 0.802 | -11.5 [-13.8, -9.3] | -9.1 [-11.2, -6.7] | yes |
| strongA_poorsep | 20 | 46.5 [43.3, 49.5] | 8.2 [4.0, 12.2] | 0.006 | 14.5 [12.9, 16.0] | 18.0 [17.0, 18.9] | yes |

## haiku-5.5 · case_v2 · multiturn

Control final P: 46.0 [43.7, 48.4] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 19 | 42.6 [40.7, 44.7] | -3.3 [-6.4, -0.2] | 0.143 | -2.5 [-3.6, -1.4] | -1.9 [-2.7, -1.0] | NO |
| noteB | 20 | 47.8 [45.2, 50.3] | 1.8 [-1.7, 5.2] | 0.679 | 0.5 [-0.5, 1.4] | -0.6 [-1.8, 0.6] | NO |
| glossA | 19 | 53.8 [50.7, 56.8] | 7.8 [4.0, 11.7] | 0.006 | 5.8 [4.4, 7.0] | 0.6 [-0.6, 1.8] | yes |
| glossB | 20 | 31.7 [30.4, 33.1] | -14.3 [-17.0, -11.6] | 0.001 | -13.1 [-14.1, -12.0] | -4.9 [-6.2, -3.4] | yes |
| orderA | 20 | 39.8 [38.3, 41.2] | -6.2 [-9.1, -3.5] | 0.001 | -0.4 [-1.9, 0.9] | 10.2 [5.9, 13.7] | NO |
| orderB | 19 | 56.7 [54.2, 59.0] | 10.8 [7.4, 14.1] | 0.001 | -2.1 [-3.8, -0.5] | -6.1 [-8.0, -3.7] | NO |
| lastA | 20 | 54.2 [48.9, 58.9] | 8.2 [2.5, 13.5] | 0.040 | -8.2 [-10.2, -6.3] | -0.1 [-1.4, 1.0] | yes |
| lastB | 20 | 44.9 [41.7, 48.1] | -1.1 [-5.1, 2.8] | 0.679 | 0.5 [-0.8, 1.8] | 0.5 [-0.5, 1.5] | yes |
| dripA | 20 | 52.9 [50.5, 55.2] | 6.9 [3.6, 10.1] | 0.004 | 4.5 [3.4, 5.6] | 1.0 [0.1, 1.9] | yes |
| dripB | 20 | 35.4 [33.5, 37.4] | -10.6 [-13.6, -7.5] | 0.001 | -9.0 [-10.8, -7.2] | -1.6 [-3.0, -0.1] | yes |
| strongA | 20 | 40.0 [38.0, 42.2] | -6.0 [-9.1, -2.9] | 0.006 | 0.2 [-1.1, 1.4] | 9.9 [8.9, 10.8] | NO |
| strongB | 20 | 52.0 [50.1, 53.9] | 6.0 [3.0, 9.0] | 0.006 | -3.4 [-4.9, -1.9] | 0.0 [-2.1, 2.2] | NO |
| strongA_poorsep | 20 | 41.6 [40.0, 43.4] | -4.3 [-7.2, -1.4] | 0.040 | 1.9 [0.9, 3.0] | 10.5 [9.9, 11.1] | NO |

## opus-4.8 · case_v2 · multiturn

Control final P: 50.6 [48.2, 52.4] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 5 | 45.2 [44.4, 46.2] | -5.4 [-7.4, -2.8] | 0.149 | -3.0 [-3.8, -2.1] | 0.0 [0.0, 0.0] | NO |
| noteB | 5 | 51.6 [49.0, 53.8] | 1.0 [-2.0, 4.2] | 1.000 | -0.3 [-1.5, 0.5] | 0.0 [0.0, 0.0] | NO |
| glossA | 5 | 56.6 [54.6, 58.6] | 6.0 [3.2, 9.0] | 0.149 | 4.3 [3.5, 5.4] | 0.0 [0.0, 0.0] | yes |
| glossB | 5 | 40.6 [39.2, 42.2] | -10.0 [-12.6, -7.2] | 0.088 | -6.8 [-7.5, -5.8] | -0.4 [-1.2, 0.0] | yes |
| orderA | 5 | 45.6 [44.4, 46.8] | -5.0 [-7.4, -2.4] | 0.185 | 0.3 [-0.4, 1.2] | 5.0 [5.0, 5.0] | NO |
| orderB | 5 | 58.6 [56.6, 60.6] | 8.0 [5.2, 11.0] | 0.088 | -3.2 [-4.1, -2.2] | -6.2 [-7.4, -5.0] | NO |
| lastA | 5 | 56.8 [55.2, 58.6] | 6.2 [3.6, 9.0] | 0.088 | -7.0 [-7.6, -6.3] | 0.0 [0.0, 0.0] | yes |
| lastB | 5 | 47.6 [45.8, 50.4] | -3.0 [-6.0, 0.6] | 0.756 | 2.8 [1.9, 4.4] | 0.0 [0.0, 0.0] | yes |
| dripA | 5 | 48.6 [47.0, 50.4] | -2.0 [-4.6, 0.8] | 0.839 | -0.7 [-1.8, 0.7] | 0.0 [0.0, 0.0] | NO |
| dripB | 5 | 45.0 [44.4, 45.6] | -5.6 [-7.6, -3.2] | 0.149 | -5.1 [-5.4, -4.8] | 0.0 [0.0, 0.0] | yes |
| strongA | 5 | 44.6 [43.2, 45.8] | -6.0 [-8.4, -3.4] | 0.172 | -0.6 [-1.2, 0.1] | 5.0 [5.0, 5.0] | NO |
| strongB | 5 | 51.6 [49.8, 52.8] | 1.0 [-1.6, 3.8] | 1.000 | -5.1 [-5.6, -4.5] | -6.4 [-7.6, -5.2] | NO |
| strongA_poorsep | 5 | 46.6 [45.2, 47.8] | -4.0 [-6.4, -1.4] | 0.245 | 0.5 [-0.6, 1.4] | 5.0 [5.0, 5.0] | NO |

## opus-5.5 · case_employment · multiturn

Control final P: 42.9 [42.0, 43.6] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 38.5 [37.6, 39.2] | -4.4 [-5.6, -3.3] | 0.001 | -4.1 [-4.7, -3.6] | -0.2 [-0.5, 0.0] | NO |
| noteB | 20 | 47.3 [46.1, 48.4] | 4.4 [3.0, 5.8] | 0.001 | 3.6 [3.0, 4.2] | 0.0 [0.0, 0.0] | NO |
| glossA | 20 | 39.2 [38.5, 40.0] | -3.6 [-4.7, -2.5] | 0.001 | -3.3 [-3.9, -2.8] | 0.4 [0.1, 0.8] | NO |
| glossB | 20 | 37.3 [36.4, 38.3] | -5.6 [-6.8, -4.3] | 0.001 | -5.4 [-5.9, -4.7] | -0.1 [-0.3, 0.0] | yes |
| orderA | 20 | 35.0 [34.4, 35.7] | -7.9 [-8.9, -6.8] | 0.001 | 5.6 [5.1, 6.1] | 11.6 [11.2, 11.9] | NO |
| orderB | 20 | 53.6 [52.2, 54.9] | 10.8 [9.2, 12.2] | 0.001 | -7.0 [-7.6, -6.4] | -7.8 [-8.2, -7.3] | NO |
| lastA | 20 | 59.5 [58.6, 60.5] | 16.6 [15.4, 17.9] | 0.001 | -9.5 [-9.9, -9.0] | 0.0 [0.0, 0.0] | yes |
| lastB | 20 | 35.3 [34.5, 36.1] | -7.6 [-8.8, -6.5] | 0.001 | 6.4 [6.0, 6.9] | 0.0 [0.0, 0.0] | yes |
| dripA | 20 | 35.4 [34.6, 36.0] | -7.5 [-8.6, -6.5] | 0.001 | -6.6 [-7.2, -5.9] | 0.0 [0.0, 0.0] | NO |
| dripB | 20 | 47.0 [46.1, 48.1] | 4.1 [2.9, 5.5] | 0.001 | 2.8 [2.1, 3.4] | -0.6 [-1.1, -0.1] | NO |
| strongA | 20 | 31.1 [30.1, 32.2] | -11.8 [-13.0, -10.4] | 0.001 | 1.6 [0.8, 2.5] | 10.2 [9.8, 10.6] | NO |
| strongB | 20 | 57.4 [56.5, 58.2] | 14.5 [13.3, 15.6] | 0.001 | -5.8 [-6.4, -5.2] | -8.0 [-8.4, -7.6] | NO |
| strongA_poorsep | 20 | 31.8 [30.8, 32.8] | -11.1 [-12.4, -9.8] | 0.001 | 2.4 [1.7, 3.1] | 11.3 [10.9, 11.7] | NO |

## opus-5.5 · case_insurance · multiturn

Control final P: 28.9 [27.1, 30.8] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 27.1 [25.1, 29.1] | -1.8 [-4.5, 0.9] | 0.531 | -1.1 [-2.6, 0.3] | 1.6 [1.0, 2.2] | NO |
| noteB | 20 | 40.6 [37.4, 43.9] | 11.7 [7.9, 15.4] | 0.001 | 6.2 [3.8, 8.4] | 0.0 [0.0, 0.0] | NO |
| glossA | 20 | 31.1 [29.6, 32.9] | 2.2 [-0.2, 4.6] | 0.460 | 1.0 [-0.1, 2.4] | 3.7 [3.0, 4.4] | yes |
| glossB | 20 | 25.9 [24.4, 27.6] | -3.0 [-5.4, -0.6] | 0.126 | -0.3 [-1.2, 0.7] | 0.0 [0.0, 0.0] | yes |
| orderA | 20 | 17.4 [16.2, 18.6] | -11.5 [-13.7, -9.4] | 0.001 | -0.4 [-2.3, 1.4] | 18.6 [17.8, 19.4] | NO |
| orderB | 20 | 31.1 [28.9, 33.5] | 2.2 [-0.7, 5.2] | 0.531 | -13.1 [-14.5, -11.6] | -11.4 [-12.4, -10.6] | NO |
| lastA | 20 | 46.0 [41.5, 50.4] | 17.1 [12.4, 21.7] | 0.001 | -1.5 [-3.3, 0.2] | 0.0 [0.0, 0.0] | yes |
| lastB | 20 | 23.1 [21.4, 24.6] | -5.9 [-8.3, -3.5] | 0.001 | 12.1 [11.2, 12.8] | 0.0 [0.0, 0.0] | yes |
| dripA | 20 | 27.1 [25.2, 29.2] | -1.8 [-4.5, 0.9] | 0.531 | -0.8 [-2.1, 0.6] | 5.0 [5.0, 5.0] | NO |
| dripB | 20 | 36.6 [34.3, 39.0] | 7.7 [4.7, 10.6] | 0.001 | 4.5 [3.1, 5.9] | 0.0 [0.0, 0.0] | NO |
| strongA | 20 | 17.8 [16.6, 19.2] | -11.2 [-13.4, -8.9] | 0.001 | 0.2 [-0.8, 1.3] | 15.2 [14.6, 15.8] | NO |
| strongB | 20 | 36.0 [33.0, 38.8] | 7.0 [3.7, 10.5] | 0.002 | -4.1 [-6.1, -2.3] | 0.7 [-0.8, 2.2] | NO |
| strongA_poorsep | 20 | 20.6 [19.2, 22.1] | -8.3 [-10.6, -6.1] | 0.001 | 2.4 [1.1, 3.7] | 16.6 [15.9, 17.2] | NO |

## opus-5.5 · case_v2 · multiturn

Control final P: 52.9 [51.9, 53.9] (n=20)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 20 | 50.7 [49.5, 51.9] | -2.2 [-3.8, -0.6] | 0.026 | -0.6 [-1.2, 0.1] | 0.0 [0.0, 0.0] | NO |
| noteB | 20 | 55.9 [55.4, 56.5] | 3.0 [1.9, 4.1] | 0.001 | 1.1 [0.7, 1.5] | -0.4 [-0.8, -0.1] | NO |
| glossA | 20 | 45.1 [44.5, 45.8] | -7.8 [-8.9, -6.6] | 0.001 | -1.8 [-2.4, -1.3] | 0.4 [0.1, 0.8] | NO |
| glossB | 20 | 53.2 [52.2, 54.2] | 0.4 [-1.1, 1.8] | 0.676 | -2.5 [-3.1, -2.0] | -2.7 [-3.0, -2.4] | NO |
| orderA | 20 | 48.4 [47.6, 49.1] | -4.5 [-5.8, -3.3] | 0.001 | 1.4 [1.2, 1.7] | 8.0 [7.8, 8.0] | NO |
| orderB | 20 | 61.0 [60.0, 61.9] | 8.1 [6.7, 9.5] | 0.001 | -7.2 [-7.9, -6.5] | -10.7 [-11.1, -10.3] | NO |
| lastA | 20 | 56.0 [55.1, 56.8] | 3.1 [1.8, 4.4] | 0.001 | -11.9 [-12.3, -11.4] | 0.0 [0.0, 0.0] | yes |
| lastB | 20 | 49.2 [47.9, 50.5] | -3.7 [-5.4, -2.1] | 0.001 | 2.4 [2.0, 2.9] | 0.0 [0.0, 0.0] | yes |
| dripA | 20 | 44.9 [43.8, 46.1] | -8.0 [-9.5, -6.4] | 0.001 | -2.1 [-3.0, -1.3] | 0.6 [0.2, 1.0] | NO |
| dripB | 20 | 56.3 [55.4, 57.1] | 3.4 [2.1, 4.7] | 0.001 | -1.3 [-1.8, -0.8] | -1.9 [-2.5, -1.5] | NO |
| strongA | 20 | 39.4 [38.6, 40.1] | -13.5 [-14.8, -12.2] | 0.001 | -3.3 [-3.6, -2.9] | 6.3 [6.0, 6.7] | NO |
| strongB | 20 | 61.5 [60.7, 62.3] | 8.6 [7.4, 9.9] | 0.001 | -5.6 [-6.1, -5.0] | -8.8 [-9.2, -8.4] | NO |
| strongA_poorsep | 20 | 40.6 [39.8, 41.5] | -12.3 [-13.6, -11.0] | 0.001 | -2.4 [-2.8, -2.0] | 6.8 [6.5, 7.0] | NO |

## sonnet-5.5 · case_v2 · multiturn

Control final P: 46.6 [45.4, 47.8] (n=5)

| arm | n | final P | Δ final vs control | p (Holm) | mean drift | step-1 gap | sign ok |
|---|---|---|---|---|---|---|---|
| noteA | 5 | 45.8 [43.6, 48.0] | -0.8 [-3.2, 1.8] | 1.000 | 0.8 [0.3, 1.4] | 3.6 [2.8, 4.4] | NO |
| noteB | 5 | 49.4 [47.0, 51.2] | 2.8 [0.4, 5.0] | 0.958 | 1.8 [1.2, 2.5] | 1.4 [-0.6, 3.4] | NO |
| glossA | 5 | 47.2 [43.8, 50.6] | 0.6 [-3.0, 4.4] | 1.000 | 0.8 [-0.4, 2.1] | 4.0 [3.2, 4.4] | yes |
| glossB | 5 | 38.2 [36.8, 40.2] | -8.4 [-10.4, -6.2] | 0.088 | -11.3 [-11.9, -10.3] | -6.0 [-6.8, -5.6] | yes |
| orderA | 5 | 44.6 [43.2, 46.0] | -2.0 [-3.8, -0.2] | 0.958 | 4.6 [3.3, 5.9] | 16.4 [16.4, 16.4] | NO |
| orderB | 5 | 55.8 [54.8, 56.8] | 9.2 [7.6, 10.8] | 0.088 | -7.6 [-8.6, -6.9] | -6.4 [-7.2, -5.6] | NO |
| lastA | 5 | 47.6 [43.6, 50.4] | 1.0 [-3.2, 4.2] | 1.000 | -13.3 [-15.5, -11.2] | 1.6 [-0.0, 3.2] | yes |
| lastB | 5 | 46.4 [43.2, 49.6] | -0.2 [-3.6, 3.0] | 1.000 | 5.1 [4.0, 6.2] | 2.0 [-0.0, 4.0] | yes |
| dripA | 5 | 43.0 [40.4, 45.8] | -3.6 [-6.4, -0.6] | 0.958 | -0.3 [-1.8, 1.0] | 4.4 [4.4, 4.4] | NO |
| dripB | 5 | 41.2 [40.4, 42.0] | -5.4 [-6.8, -4.0] | 0.088 | -9.1 [-10.5, -7.8] | -4.4 [-5.2, -3.6] | yes |
| strongA | 5 | 40.0 [38.8, 41.2] | -6.6 [-8.2, -5.0] | 0.088 | 2.7 [1.9, 3.7] | 13.2 [12.4, 14.0] | NO |
| strongB | 5 | 45.2 [43.2, 47.8] | -1.4 [-3.8, 1.4] | 1.000 | -14.1 [-15.4, -12.4] | -9.4 [-10.6, -8.2] | yes |
| strongA_poorsep | 5 | 40.0 [38.4, 41.6] | -6.6 [-8.6, -4.6] | 0.088 | 2.3 [1.0, 3.4] | 14.4 [14.4, 14.4] | NO |

## Unparseable runs

- fable-5.1 / glossA: 1
- gpt-5.6-luna / dripA: 2
- gpt-5.6-luna / dripB: 1
- gpt-5.6-luna / strongB: 2
- gpt-5.6-terra / dripA: 3
- gpt-5.6-terra / dripB: 2
- gpt-5.6-terra / glossA: 3
- gpt-5.6-terra / lastA: 1
- gpt-5.6-terra / noteA: 1
- gpt-5.6-terra / noteB: 1
- gpt-5.6-terra / strongA_poorsep: 1
- gpt-5.6-terra / strongB: 2
- gpt-6-luna / control: 3
- gpt-6-luna / dripA: 3
- gpt-6-luna / dripB: 4
- gpt-6-luna / glossA: 7
- gpt-6-luna / glossB: 3
- gpt-6-luna / lastA: 1
- gpt-6-luna / lastB: 3
- gpt-6-luna / noteA: 6
- gpt-6-luna / noteB: 4
- gpt-6-luna / orderA: 4
- gpt-6-luna / orderB: 4
- gpt-6-luna / strongA: 6
- gpt-6-luna / strongA_poorsep: 1
- gpt-6-luna / strongB: 6
- haiku-5.5 / glossA: 1
- haiku-5.5 / glossB: 1
- haiku-5.5 / lastA: 2
- haiku-5.5 / noteA: 1
- haiku-5.5 / orderB: 1
- haiku-5.5 / strongA: 1