# Bash SFT note 02 — cumulative Session 2 report

Date: 2026-08-21
Status: research note, not a paper
Correction: fair-base comparison added 2026-08-21

## Result

Execution-backed SFT taught a 1.5B model the raw-command interface and the
generator-matched file-retrieval distribution. The best observed arm passed
88/98 locked in-domain components. It did not pass the one post-selection,
generator-independent historical canary. Broader Bash transfer and any win over
`distil-lfm25-shellper` remain unproven.

## Fair-base correction

The original strict comparison gave stock 0/98 and Arm 2 88/98. Its prompt
explicitly required exactly one raw command with no prose or Markdown fence, so
stock did fail a stated product instruction. But it supplied no format
demonstration and did not disclose the judge's exact command-policy allowlist
and restrictions, while Arm 2 had repeatedly learned the target shape during
SFT. We retain the original strict result for product-contract auditing, but no
longer use 0/98 as the semantic stock baseline.

First, a deterministic whole-response single-command Markdown adapter was
applied symmetrically to the original stock, prior-S2, and Arm-2 completions.
It removes only a single outer Markdown fence around a whole-response command;
it does not choose a snippet from prose or rewrite shell. The unchanged policy
and execution judge then produced:

| original completions, adapted symmetrically | stock | previous S2 | arm 2 |
|---|---:|---:|---:|
| original raw one-line contract | 0/98 | 46/98 | 98/98 |
| whole-response command accepted | 98/98 | 97/98 | 98/98 |
| policy, syntax, and execution accepted | 69/98 | 84/98 | 98/98 |
| execution pass | **9/98 (9.2%)** | **10/98 (10.2%)** | **88/98 (89.8%)** |

Arm 2 minus stock is +80.6 percentage points, with a paired
component-bootstrap 95% interval of +71.4 to +88.8 points (10,000 resamples,
seed 42).

Second, pinned stock and pinned Arm 2 were regenerated with byte-identical
effective prompts that disclosed the exact output format and allowed-command
policy. The shared context included one neutral user-to-assistant format
no-op demonstration whose assistant answer was `true`; it did not demonstrate a
file-retrieval solution. Each model ran on a separate NVIDIA RTX 6000 Ada
Generation GPU with greedy decoding and seed 42. No adapter was needed.

| identical contract-calibrated generation | stock | arm 2 |
|---|---:|---:|
| raw one-line contract | 98/98 | 98/98 |
| policy accepted | 86/98 | 98/98 |
| syntax valid and executed | 85/98 | 98/98 |
| execution pass | **8/98 (8.2%)** | **89/98 (90.8%)** |
| exact gold command | 0/98 | 80/98 |

Arm 2 minus stock is +82.7 percentage points, with a paired
component-bootstrap 95% interval of +74.5 to +89.8 points (10,000 resamples,
seed 42). Both temporary generation droplets were deleted after aggregate
artifacts were pulled.

The corrected conclusion is narrower and stronger: stock is a low, non-zero
semantic baseline once the interface disadvantage is removed, while Arm 2
retains an approximately 81–83 point in-domain execution advantage. This is
still a within-generator selection set and does not establish broad transfer.

## Since Session 1

Session 1 improved imitation while its real probes stayed at 0/3. Session 2
first ran a small method pilot: 164 assembled rows, 137 train / 27 validation,
one response-only epoch from stock. On 500 historical imitation examples, exact
Bash moved from 61/500 (12.2%) to 70/500 (14.0%). Bash-token F1 moved from 41.5%
to 43.8%. Cloud exact stayed 0/9. This was text imitation, not execution.

The latest run replaced exit-status labels with fixtures and outcome checks.
Sixteen execution-backed seeds preceded a synthetic 3,000-row candidate batch
that the project describes as trace-inspired. The public packet does not map
rows back to source archetypes. Every final row version was independently run on
macOS and required the expected exit code, exact stdout/stderr, and no
filesystem change. Ten documented repairs were applied before the recorded
promotion execution.

Grouped splitting produced 2,402 train, 297 validation, and 301 test rows over
998 connected components. The locked selection set contains one representative
from each of 98 test components. Exact normalized-command overlap across splits
is zero. Generator templates and vocabulary remain shared, and no semantic-near-
duplicate or source-time audit was completed.

## Four arms

All arms began at the same pinned stock
`Qwen/Qwen2.5-Coder-1.5B-Instruct` revision and used response-only full SFT,
bf16, batch 4, gradient accumulation 4, maximum length 1,024, seed 42, and
greedy evaluation.

| arm | rows | epochs | LR | eval loss | execution | wall |
|---|---:|---:|---:|---:|---:|---:|
| arm 1, full | 2,402 | 1 | 2e-5 | .0262 | 84/98 | 60.3 s |
| arm 2, full | 2,402 | 2 | 2e-5 | .0213 | **88/98** | 113.7 s |
| arm 3, lower LR | 2,402 | 1 | 1e-5 | .0415 | 79/98 | 61.7 s |
| arm 4, representatives | 802 | 1 | 2e-5 | .0544 | 71/98 | 20.2 s |

Arm 2 minus Arm 1 is +4.1 percentage points, paired component-bootstrap 95%
interval 0.0 to +9.2 points (10,000 resamples). It is the best observed recipe,
not proof that two epochs always win. Arm 4 also received fewer rows and
optimizer updates, so it cannot isolate the causal value of variants.

Arm 2 exactly matched 80 gold commands but passed 88 fixtures. It achieved
98/98 raw-format, policy, syntax, execution, and immutability checks. Its ten
semantic failures were four exit mismatches and six stdout mismatches, with no
timeouts or mutations. Fourteen of twenty small families were perfect; the
weakest were `jq_json_filter` (2/4), `rg_files_candidates` (3/6), and
`rg_regex_hidden_exclude` (2/4); all three rates were 50%.

## Original strict base-model comparison

| same 98 prompts | stock | previous S2 | arm 2 |
|---|---:|---:|---:|
| raw one-line contract | 0/98 | 46/98 | 98/98 |
| policy accepted | 0/98 | 0/98 | 98/98 |
| strict execution pass | 0/98 | 0/98 | 88/98 |
| exact gold command | 0/98 | 0/98 | 80/98 |

Stock wrapped all completions in fences or prose, so the strict product scorer
did not execute them. This remains a failure under the requested direct-command
contract, but 0/98 is not a clean estimate of stock Bash semantics. The dated
correction above supplies both symmetric extraction and identical
contract-calibrated generation results.

## External canary

The historical handoff-search canary was reconstructed after arm selection. It
requires translating “current directory” to `.`, pruning generated trees,
handling spaces safely, matching three filename variants, and returning a
newest-first long listing. Stock, previous S2, and all four new arms scored 0/1.

Arm 2 recovered the filename variants and much of the pruning intent, but
searched a literal directory named `current directory` and omitted NUL-safe
long-listing/time-order composition. This is a qualitative transfer alarm, not
a rate estimate.

## Limits

- The 98 tasks are a within-generator selection/development set, not an
  untouched final test.
- The generator process read only prompts, but locked gold artifacts existed
  elsewhere on the remote filesystems. Future generation hosts must receive
  prompts only.
- The scorer used lexical rejection, a restricted `PATH`, limits, timeouts, and
  full snapshots because the managed macOS host could not apply its kernel
  sandbox profile. It is not a hostile-shell security boundary.
- The run used a dirty, uncommitted source worktree. This public packet freezes
  aggregate evidence, not a hermetic reproduction environment.
- The fair-evaluation packet publishes aggregate counts and paired intervals,
  not prompts, completions, fixtures, machine details, or execution records.
- The operator record says a timed four-arm transfer follow-up expired during
  environment provisioning and the four temporary GPUs were deleted. No durable
  training log or checkpoint exists, so no model result is claimed.

## Next experiment

Freeze at least 20–50 generator-independent executable tasks first. Create
unrelated training rows for deictic roots and multi-constraint composition.
Compare a fresh stock run against low-learning-rate continuations from Arm 2 at
matched update budgets. Select on the new external suite and retain the current
98 tasks only as an in-domain regression gate. Add an AST-aware or OS-enforced
sandbox before untrusted use.

The next win is external-task transfer without losing Arm 2's 100% raw-command,
policy, syntax, and immutability rates—not a higher score on the current 98.