Bash SFT note 03: file search reached 92/98, but more data did not help

30 August 2026 · Session 3 report · research note, not a paper · follows note 02

We ran two experiments on the same 1.5-billion-parameter coding model. The first asked what kind of training examples help it imitate Bash answers. The second asked whether its file-search commands actually work on macOS. The best file-search checkpoint passed 92 of 98 held-back tasks on its first try. Adding more unverified traces or repeating the hardest training families did not improve that result. In a separate matched-prompt comparison, that checkpoint passed 90 tasks, its exact stock base passed 8, and a small natural-language-to-shell specialist passed 4.

Scale mix A on many Adas, or test in real life first?

Test the 92/98 checkpoint in real life first. More GPU time is justified only after the model works on searches someone actually needed. When training resumes, add newly executed examples rather than more copies of the same failures: both the unverified harvest data and the hard-family repetition have already failed to improve one-try accuracy.

1. The product must return one safe, read-only macOS Bash command for a file-search request. The broader imitation study includes other kinds of Bash work, so Mix A’s win does not establish product quality.
2. The harvest data enlarged the product training mix by 50%, but 91/98 stayed 91/98. Tripling the hard-family examples also failed to beat the simpler GFR-only checkpoint.
3. Six locked tasks still fail across five families: rg_files_candidates, rg_count, rg_regex_hidden_exclude, jq_json_filter, and find_directories. Three commands return the wrong exit status; three return the wrong text. Real searches in those families will show whether the gap matters in use.
4. Next, execution-check the 218 harvest candidates still in quarantine. Only then consider renting several Ada GPUs for genuinely different recipes.

What the arm names mean

An “arm” is one version of the training recipe. This experiment changed only the kinds of examples included in supervised fine-tuning (SFT): the model sees a request and a desired answer, then adjusts its weights to make that answer more likely. The base model, number of optimizer steps, and other settings stayed fixed, so the comparison isolates the value of task solutions, repairs, and trajectories.

Shared setup: pinned Qwen/Qwen2.5-Coder-1.5B-Instruct revision, response-only full SFT, 561 optimizer steps, batch 16, max length 1,024, learning rate 2e-5, seed 42, and one 48 GB RTX 6000 Ada.

Task solution = one natural-language request to one command. Error repair = a failed command and the fix. Trajectory = a multi-step trace. Exact % is greedy string match on the 1,030-row episode validation set, not GFR locked execution.

What GFR and harvest mean

GFR means Grok File Retrieval, the product dataset. Each example pairs a natural-language file-search request with one Bash command. Unlike an ordinary text benchmark, the harness actually runs the command on a synthetic directory, then checks its exit status, output, error output, and whether any file changed. The split is 2,402 training tasks, 297 validation tasks, and 98 locked evaluation tasks.

Locked evaluation means those 98 tasks were kept out of the training rows. They were still used as a development scoreboard while recipes were compared. Pass@1 means the model gets one deterministic attempt at each task; 92/98 is 93.9% on that set. “Locked” limits direct training leakage, but it does not make the score an untouched final test or independent of the generator that created the rest of GFR.

Harvest is a 30 August 2026 scrape of 1,201 commands that looked like file search in real Mini coding traces. Those commands were not run through the fixture harness, so their correctness was unknown. Mixing them with the 2,402 GFR rows increased the training set to 3,603 rows but did not raise pass@1.

Episode mix is the separate, older general-Bash corpus of 8,970 rows used for Arms A–D. It measures imitation, not executable retrieval. Weight decay is a training constraint that discourages weights from growing too large; the best recipe here used 0.1, but this small study does not prove that weight decay alone caused the extra pass.

Two separate contests, not one winner

The cleanest summary is one question and one result per experiment:

Arm A did not “win GFR.” It won the mix-composition study. The product checkpoint is last40 wd=0.1 at 92/98. More general imitation data helps the product only if it transfers to executable file-search, which this report has not shown.

Locked file-search: 88/98 to 92/98

The progression began with Note 02’s Arm 2: two passes through the 2,402 GFR training rows at a 2e-5 learning rate, scoring 88/98. One additional epoch at the gentler 1e-5 rate reached 91/98. Adding harvest data left the score unchanged. The kept model then trained for one more GFR-only epoch with the same low rate, a cosine schedule, and weight decay 0.1, reaching 92/98.

Several plausible alternatives failed. A third epoch at the original, higher learning rate had already been discarded. Repeating the remaining hard families did not help. Starting from stock with cosine restarts fell sharply, and a recovery run remained below the best checkpoint. A final Kimi K3-suggested hard-family continue tied 92/98 but needed more repeated data, so the simpler GFR-only checkpoint stayed.

last40 wd=0.1 is the kept checkpoint. All 98 outputs remained one-line, policy-compliant, syntactically valid, and read-only. The six failures split evenly between a wrong exit status and wrong output. Because one additional pass on a 98-task set is a small difference, real-use testing matters more than the apparent one-point lead.

Sample outputs to eyeball

These examples make the aggregate score concrete. The model received each shortened request once and returned one command. “Gold” is the expected command when the model’s version failed. All names come from synthetic fixtures, not private paths.

Passes (92/98)

Misses (6/98)

The six remaining errors are mostly small command details with large behavioral consequences: a missing exclusion, an incomplete file extension, a literal match where a word boundary was required, or a case-sensitive directory test. The model still obeyed the one-line and read-only contract on every miss; it simply returned the wrong result or exit status.

Full commands for the six misses are in samples.json. Format/policy/syntax still passed on every miss. The Kimi hardfam run also scored 92/98; it kept README-case instead of the piped-YAML miss.

Live probes, 30 August 2026

Six informal searches were also tried on the Mac with the earlier 91/98 harvest-continue checkpoint. Three were useful. The failures reveal a more practical weakness than the synthetic score: the model mishandled phrases such as “this directory” and placed find pruning logic in the wrong order. These probes are diagnostic examples, not a scored test set.

Named fixture roots are strong. “This directory” / “current directory” and find-prune order are not. That is the real-life gap before any Ada scale-up. No GPU boxes were created for these probes; none were left running.

The harvest mix in numbers

The harvest experiment tested a tempting idea: perhaps more commands from real coding sessions would be enough, even before each command was verified. It was not. Quantity increased; locked execution accuracy did not.

• GFR train 2,402 + harvest file-search 1,201 = 3,603 train rows. Trainer val stayed GFR val (297). Locked 98 stayed out of train.
• Harvest slice: 472 task solutions, 427 error repairs, 302 trajectories. Not execution-backed. 218 further candidates remain quarantined pending execution.
• A 2048-token continue and a constant 1.5e-5 follow-up started after the 91/98 scores. Those GPUs were torn down before those checkpoints were pulled. They are not results in this packet.

Last window: 92/98, then Kimi tied it

After the 91/98 harvest continue, four more recipes ran on RTX 6000 Ada GPUs. A Kimi K3 consult then proposed one final recipe. Only the GFR-only run with weight decay 0.1 added a pass. Kimi’s follow-up tied it, so there was no reason to keep the more complicated model. Every checkpoint reported here was downloaded and hashed before its GPU was deleted.

The hardfam mix contains the 2,402 GFR rows plus three extra copies of 492 executed examples from four weak families: find_directories, rg_count, rg_files_candidates, and rg_regex_hidden_exclude. Unlike harvest, these extra rows were verified by execution. Even so, repeating them did not push the score beyond 92/98. The result tells us repetition was not enough; it does not establish why.

Fair baselines: 90/98 versus 8/98 and 4/98

The useful comparison is not “our interface versus somebody else’s interface.” It is the same request, the same response contract, and the same scorer given to models that natively return shell commands. We therefore reran three checkpoints: the kept model, the exact stock model from which its training lineage began, and a smaller shell specialist chosen before seeing its outputs.

The specialist is AryaYT/nl2shell-0.8b, a 0.9-billion-parameter Qwen3.5 model. Its model card says it was tuned on 8,130 natural-language-to-Bash pairs plus 40 synthetic macOS examples. That makes it a plausible specialist baseline, not a claim that its training contract matches GFR.

What the gap means. All three models returned one raw line for every request, so formatting no longer explains the result. The stock model usually produced plausible-looking commands, but exact output and exit-status semantics defeated it. The specialist often assumed absolute or home paths and used commands outside this read-only macOS contract. The sandbox blocked unsafe commands; one allowed command changed its disposable fixture and was caught as a failure. “Trained for shell” is not the same as “trained for this shell contract.”

What was held equal. The three runs used byte-identical message content: the request, the full fixture-blind evaluator contract, and one neutral formatting example. Each model used its own tokenizer and chat template, then greedy decoding with seed 42, a 1,024-token input limit, and a 256-token output limit on the same RTX 6000 Ada. Predictions were pulled and hashed before that GPU was deleted. Fixtures never went to the GPU; strict execution happened afterward on macOS Bash 3.2.

Why 90 here but 92 above? The frozen primary result used the product prompt and remains 92/98. This comparison adds a longer contract explanation so unfamiliar public models are not penalized for hidden rules; under that matched prompt, the kept checkpoint passes 90/98. Do not average the two numbers: 92 describes the selected product run, while 90 is the like-for-like baseline run. Against stock, the matched gain is 83.7 percentage points (95% paired component-bootstrap interval: +75.5 to +90.8 points). The specialist is 4.1 points below stock, with an interval from −10.2 to +2.0 points.

Possible next steps

The next stage should reduce uncertainty about real use, not add another nearby training score.

1. Real-life gate. Run the last40 wd=0.1 checkpoint on live Mini file-search, including the six remaining locked families and the historical handoff canary. Do not train on the canary.
2. Execute, then scale. Promote the 218 quarantined harvest candidates only after macOS execution. Do not upsample the current six misses again.
3. Then many Adas. New LRs and schedules on that larger execution-backed mix. Do not rerun packed SFT, LoRA, 3B, repair-only 5e-6, extra 2e-5 epochs, cosine_with_restarts, or another hardfam copy of the same misses.
4. Keep GFR 98 as a regression gate. Select on real-use tasks frozen before the next data drop. Never publish training JSONL.

Next success criterion: the 92/98 model is useful on a live search the operator actually needed, without dropping contract rates, before we pay for a GPU farm.

Evidence and reproducibility

The public evidence packet contains aggregate metrics, selected sanitized examples, and hashes. It excludes training JSONL, private traces, prompts with personal paths, full completion dumps, fixtures, credentials, and weight tensors.

• summary.json
• samples.json
• four-regime.json
• locked-eval.json
• last40.json
• fair-baselines.json
• report.md
• SHA256SUMS

Kept checkpoint: last40 wd=0.1, Qwen2.5-Coder-1.5B-Instruct full SFT, GFR 2,402 rows × 1 epoch, LR 1e-5, cosine, weight decay 0.1, max length 1,024. Unpublished model.safetensors SHA-256 b3558c0f140a659f0c661bf9795ed59049cdfed5db42c1e3e8e8480e8ad901b2. Train wall 57.3 s; peak 15.22 GiB on one NVIDIA RTX 6000 Ada Generation GPU. Kimi hardfam weights (not kept) SHA-256 e57515db06eb34a0a70ca33ddf2c9a23255206b492477e25c8c4ecad3cc56d41.

Bottom line

For general Bash imitation, keep the mixture of solutions, repairs, and trajectories. For executable file-search, the simplest winning recipe was one more low-learning-rate GFR epoch with weight decay 0.1. More unverified data and more copies of known hard cases did not help. Do not scale GPUs until the 92/98 checkpoint succeeds on real searches and the next training rows have been execution-checked.