Bash SFT note 01: a 4.5-second train, a deleted GPU, and a 1.8-point exact bump

21 August 2026 · research note, not a paper · Bash SFT series note 01 · separate from the earlier tab-title namer notes on this site

What we did, in order

1. Session one: packed then response-only SFT of 1.5B on the 4000-row mix. Teacher-forced loss 1.01 to 0.72, bash F1 21.1% to 34.5%, exact 0.5% to 7.25%. Both real probes still failed.
2. That mix stamps exit 0 as verified and randomly splits paraphrases. We kept the trainer (response-only, packing off, SDPA, bf16) and threw out the mix for this note.
3. Qwen2.5-Coder-3B-Instruct response-only lost to 1.5B epoch 3 after one and two epochs (3B e2 exact 4.5% / bash F1 32.1% / tool 49.5% vs 1.5B e3 7.25% / 34.5% / 51.0%). Scaling was not the lever. This note did not train 3B again.
4. The mix for this note is 164 assembled rows (158 task_solution + 6 error_repair, 0 trajectory), split 137 train / 27 val. Provenance: 155 curated, 9 executable-spec. Families held out of train: newest-file find, quoting, git-status, and a stubbed cloud listing.
5. One rented NVIDIA RTX 6000 Ada Generation GPU (48 GB). Train 4.51 s, peak 13.90 GiB, train_loss 0.779, eval_loss 0.733. Then we destroyed that VM before scoring.
6. Scoring needed a second rented GPU of the same type. Stock 423 s then SFT 478 s on that box (SSH wall times; those numbers are not in summary.json). We also requested three more GPUs; copying a 2.9G checkpoint onto new boxes would have been slower than scoring on one already-hot machine. Wasted request. Both GPUs were destroyed after the pull. No other machines were used.
7. Holdout is the first 500 rows of a Mini-trace validation file, not a random sample. Exact id overlap with this note’s 137-row train split: 0. No near-duplicate leak check. Scoring device in the artifact: cuda, dtype bf16, greedy, 256 new tokens.
8. A GPU status page had a JS bug (const PORT replaced with 8787, blank charts). A later fetch treated the snapshot envelope as the box list, so it looked like zero rented GPUs while one was live. Empty-state copy hid a parse bug as “no VMs.”

Stock vs SFT on 500 tasks

CUDA on the rented RTX 6000, greedy. n=500 all task_solution. Deltas are rounded from unrounded rates, so 41.5 to 43.8 is listed as +2.4 pp.

Non-cloud n=491 carries the entire 61 vs 70 exact lift. Cloud n=9: exact 0.0%/0.0%, stem 55.6%/55.6%, tool 55.6%/55.6%, bash F1 18.3%/19.8%, text F1 24.7%/49.5%. Do not cite the cloud slice as a win.

Stock SFT. Chart language follows the first PorkiCoder papers: inline SVG, no JavaScript, teal stock, green SFT, red for a fail.

What each metric means

Scoring is greedy generation plus a teacher-forced pass on the gold reply. Commands are taken from fenced ```bash blocks. Comments and extra spaces are stripped before match.

n

How many tasks were scored. Here, 500. All of them are task_solution rows.

Teacher-forced loss

How surprising the gold tokens are, given the prompt. Lower means the gold reply looks more like something this model would write. It is not “did the command work.”

Teacher-forced token acc

Share of gold tokens the model would have picked as the next token. A style clone can raise this without copying the command.

Closed bash fence

Did the reply contain a fenced bash block at all. Stock already did this on 499 of 500 rows. Formatting was not the problem.

Exact bash match

After cleanup, the text inside the fence is identical to gold. The headline number. 61 stock rows, 70 SFT rows.

First-line match

The first non-comment line of the command matches gold’s first line. Useful when a later pipe is wrong.

Command-stem match

The opening tokens of the command match, stopping at a pipe, flag, or connector. Roughly “did it start the same way.”

First-tool match

The first program name matches (find vs grep). The coarsest command metric. Stock was already at 86.4%.

Extracted-bash token F1

Token overlap inside the fence versus gold’s fence. Partial credit when the command is close but not exact. This is the honest “how much of the command text moved” number.

Full-text token F1

Token overlap of the entire reply, including the English wrapper. Inflated here because SFT always emits the same opening sentence.

SFT-style prefix

Does the reply start with Here is a copy-pasteable command. Stock: never. SFT: always. That is a wrapper lock, not skill.

What this table is saying

Read it as three stories, not one lift.

Story 1: a small copy bump. Exact 12.2% to 14.0% is nine extra rows whose fenced command matched gold after cleanup. First-line moved with it (12.4% to 14.0%). That is string match, not a passing test suite.

Story 2: the model more often grabs the right tool, then still writes the wrong command. First-tool 86.4% to 91.6% and stem 50.4% to 56.8% moved more than exact. Bash-token F1 only went 41.5% to 43.8%. The SFT is better at starting find or git and still misses flags, paths, and the rest of the line.

Story 3: the huge F1 is a costume. Full-text F1 41.4% to 66.3% tracks the style prefix going 0% to 100%. Fence rate was already 99.8%. Teacher-forced loss dropped because gold now looks like the SFT dialect. None of that says the command did the job.

Cloud n=9 stayed exact 0/0, stem 55.6/55.6, tool 55.6/55.6. The whole 61 vs 70 lift sits in the other 491 rows. One seed. No significance test. Session-one live probes remain 0/3 and were not re-run on this checkpoint.

What we are not claiming

• Executable task success. Session-one probes remain 0/3 and were not re-run.
• That the 4000-row mix is good data. It is not. Exit 0 is process success.
• A ship. 137 rows and 4.51 s of train is a method smoke.
• That extra GPUs would have helped this 500-row score.

Possible next steps

• Highest leverage. Score executable pass@1 on a small frozen task set (fixtures, expected stdout or final state, forbidden-change rules). The 1.8-point exact bump is string match against gold, not “the command did the job.”
• Highest leverage. Fix mix quality before adding rows. Stop treating shell exit 0 as verified. Hold out whole families, not shuffled paraphrases.
• Highest leverage. Near-duplicate leak check on the 500-row holdout. Exact id overlap with train is already 0; template and n-gram overlap is still unmeasured.
• Stubbed cloud eval for the n=9 slice (exact stayed 0/0). Query-format checks against recorded responses. Do not run generated cloud commands against live accounts.
• Re-run the three live probes that were 0/3 as canaries on this checkpoint, or stop citing them.
• Keep any new score on the same CUDA / bf16 / greedy setup. A laptop Metal score is a different experiment.
• Wasted motion. More rented GPUs, more epochs on unverified rows, or chasing full-text F1. That metric is the wrapper Here is a copy-pasteable command.

Evidence

Packet: ./bash-sft-01/. Per-row files keep scores only. Gold commands and model completions are not published. No weight tensors.

• summary.json
• report.md
• row_metrics.jsonl (scores only)
• train_summary.json
• mix_manifest.json
• SHA256SUMS