Bash SFT note 04: live-style training dropped 92/98 to 82/98
2 September 2026 · Session 4 report · research note, not a paper · follows note 03
Note 03 kept a 1.5-billion-parameter file-search checkpoint that passed 92 of 98 frozen macOS tasks. Live use does not talk like those tasks. We trained one continuation of that checkpoint on 2,240 shorter, conversational examples, using the same recipe that produced 92/98. On the frozen 98 it scored 82. On a new 27-item live-style set it scored 2. The model still invents a folder name instead of searching the already-selected tree with a dot. The 92/98 weights stay in the product. These new weights do not ship.
How to read this report
Two scoreboards, never mixed:
- GFR v1 is the frozen 98. Same files, same hashes, original system prompt. Report 88/94 non-newest plus 4/4 newest. The four newest items are report-only because the product already orders listings in trusted code and blocks
find -exec. - GFR v2 locked-hard is 27 new synthetic items. Short prompts, both sides of type versus substring, a named child versus the whole tree, and unsupported outcomes. A pass requires the command to work on a hidden fixture, or an exact
unsupported:<slug>line that is never executed.
92/98 remains a historical fact. 82/98 is a regression on that same set. 2/27 is the live-style result. None of these numbers comes from private laptop queries.
What held. The incumbent still scores 92/98 under the original prompt, so the freeze was not corrupted. Every continuation output stayed one line. On the frozen 98, policy, syntax, and filesystem-unchanged all remained 98/98. The Ada box was deleted after a 2.875 GiB weight pull.
What the prompt change did, before any gradient. Replacing the original system message with a product envelope, and changing nothing else, dropped the incumbent from 92/98 to 86/98. Four of those misses were absolute or home paths taken from the envelope. That is a prompt-format effect, not a training effect.
What failed to ship. Non-newest frozen accuracy fell from 88/94 to 81/94. The live-style set stayed at 2/27. Unsupported recall was 0/6: the model listed files instead of saying the tools cannot see the requested distinction. A private 13-query canary was not run, because this candidate is not selected.
Keep the 92/98 checkpoint
Do not replace the product weights. One Ada run is enough to reject this mix. The next change is the training rows, not a second starting checkpoint and not another learning-rate search.
- The product must return one safe, read-only macOS Bash command, or an explicit unsupported line, for a file-search request typed in ordinary English.
- The frozen 98 tasks are long specification prompts. Live requests are short: “html files”, “find me markdown”, “images from last 3 days”.
- Teaching those short prompts by paraphrasing old golds that still search a named folder taught the model to invent a folder when the prompt no longer named one.
- Next mix: more executed
find .examples, fewer named-root paraphrases, same 1.5B continuation recipe.
What GFR v1, v2, and unsupported mean
GFR v1 is the 2,402 / 297 / 98 file-search split from notes 02 and 03. Those files were hashed before this run and were not rewritten. The incumbent is last40 wd=0.1, Qwen2.5-Coder-1.5B-Instruct, SHA-256 b3558c0f140a659f0c661bf9795ed59049cdfed5db42c1e3e8e8480e8ad901b2.
GFR v2 is a private continuation mix. It is not in git. Train 2,240, val 117, locked-hard 27. Median train prompt 39 characters, versus 155 in GFR v1. Gold commands were executed on Darwin except for unsupported outcome lines.
Unsupported is the non-shell line unsupported:<slug>. The app parses it before command policy and does not execute it. The four frozen slugs are image content, provenance metadata, clarification, and not-a-file-search. A colon no-op is not an unsupported outcome; : is an allowed builtin.
Envelope is a short system message that states the locked folder, that . means the whole tree, and that unsupported is a non-shell line. It is a labelled ablation on the frozen 98, not a replacement for 92/98.
The 2,240-row mix
The mix was built to match live length, not to copy private laptop queries. Style gate: at least 60% of prompts at most 80 characters. This build landed 78% at most 80 characters.
| Component | Rows in train | Rule |
|---|---|---|
| Shortened GFR v1 golds | 1,243 | Same executed answer, conversational prompt. Named-root golds kept. |
| GFR v1 spec anchors | 522 | Original long prompts, so the frozen 98 template is not forgotten. |
| Dot-root augmentations | 284 | Train-only rewrites to find ., re-executed. |
| New live archetypes | 191 | Type versus substring, recency, topic, unsupported pairs. Too few. |
Shared recipe with the 92/98 run: response-only full SFT, 1 epoch, learning rate 1e-5, cosine, weight decay 0.1, batch 4, accumulation 4, max length 1,024, seed 42, one 48 GB RTX 6000 Ada. Train wall 59 s. Peak 16.60 GiB.
Locked scores
Arm 0 is the incumbent with no new gradient. The continuation is that checkpoint trained once on the v2 mix. Envelope columns use the new system message on the same 98 tasks; they do not replace 92/98.
| Run | v1 original | Non-newest | Newest (report) | v1 envelope | v2 locked-hard | Exact gold (v1 original) |
|---|---|---|---|---|---|---|
| Incumbent Arm 0 | 92/98 | 88/94 | 4/4 | 86/98 | 2/27 | 86/98 |
| Live-style continuation | 82/98 | 81/94 | 1/4 | 84/98 | 2/27 | 69/98 |
Promotion needed non-newest at least 88/94 and locked-hard cluster pass 100% with item pass at least 95%. Both failed. Format/policy/syntax on v1 original remained 98/98 for both runs. Envelope on the incumbent had 4 absolute-or-home-path refusals.
The original prompt still measures 92/98. The continuation is 82/98 on that same prompt. Envelope alone already costs the incumbent six passes.
The continuation did not move the live-style set. Both runs passed 2 of 27.
Sample outputs to eyeball
These requests are synthetic. They follow the live length and wording, not private laptop text. “Gold” is the executed command, or the unsupported line.
The two live-style passes (2/27)
| Family | Request | Model command |
|---|---|---|
| pass named child | files under the html folder | find 'html' -type f -print | sort |
| pass named child | files under the me folder | find 'me' -type f -print | sort |
Typical misses
The pattern is the path slot. A short type or recency request still becomes find 'images', find 'docs', or find 'lantern' instead of find .. Topic words become folders. Unsupported requests become broad listings.
| Why | Request | Model | Gold |
|---|---|---|---|
| path type | html files in this folder | find 'this folder' -type f \( -iname '*.html' \) -print | sort | find . -type f \( -iname '*.html' -o -iname '*.htm' \) -print | sort |
| path substring | files with html in the name | find 'web app' -type f -iname '*html*' -print | sort | find . -type f -iname '*html*' -print | sort |
| path topic | markdown related to lantern | find 'lantern' -type f \( -iname '*.md' -o -iname '*.markdown' \) -print | sort | rg -l -i 'lantern' --glob '*.md' . | sort |
| path recency | images from the last 24 hours | find 'photos' -type f -name '*.jpg' -mtime -1 -print | sort | find . -type f \( -iname '*.jpg' -o ... \) -mtime -1 -print | sort |
| refuse vision | pictures of a red bicycle | find 'bicycle pictures' -type f \( -iname '*.jpg' -o -iname '*.jpeg' \) -print | sort | unsupported:needs-image-content |
Full synthetic samples are in samples.json. No private Application Support logs are in the packet.
Why the path slot won
GFR v1 never trained a dot-rooted find. Every find gold started at a named folder. Live English often does not name a folder, because the app has already locked one. The continuation mix still contained 1,243 paraphrases whose gold kept that named folder, and only 191 new live archetypes plus 284 dot-root rewrites. A 1.5B model that has filled a path slot thousands of times will keep filling it from the nearest noun: “html”, “photos”, “lantern”, “this folder”.
The two passes are the exception that proves the rule. When the prompt explicitly named a child that existed in the fixture, the model used that child. When the prompt meant the whole locked tree, it invented a child.
Envelope text that included an absolute locked-root path made the incumbent emit that path four times. The envelope stays useful as a contract document. It is not, by itself, a fix for the path slot.
Possible next steps
Change one variable: the mix. Keep the 92/98 initialization and the 1e-5 / wd 0.1 recipe.
- Rebuild v2 so short whole-tree prompts gold
find ., executed, not handwritten. Named-root paraphrases of GFR v1 should not dominate. - Keep unsupported pairs with representable twins. The parser already exists in the app. Do not gold a colon no-op.
- Hash locked-hard before the next droplet exists. Run the incumbent under both prompts first. Then one Ada continuation.
- Pull about 2.9 GiB of
model.safetensors, record SHA-256, delete the GPU. Do not leave a powered-off GPU billing.
Next success criterion: non-newest frozen score at least 88/94, locked-hard cluster pass 100% with item pass at least 95%, and only then a private canary that can veto shipping.
Evidence and reproducibility
The public evidence packet contains aggregate metrics, sanitized synthetic samples, and hashes. It excludes training JSONL, private traces, prompts with personal paths, full completion dumps, fixtures, credentials, and weight tensors.
Kept product checkpoint remains last40 wd=0.1, unpublished SHA-256 b3558c0f140a659f0c661bf9795ed59049cdfed5db42c1e3e8e8480e8ad901b2. Rejected continuation SHA-256 303275de21da76fa08a9f602ba033e9e34ce44253031080ec1283caabdd80cf9 (3,087,467,144 bytes). Droplet 597095105 deleted after that pull. Train wall 59.0 s; peak 16.60 GiB on one NVIDIA RTX 6000 Ada Generation GPU.
Bottom line
Shortening the prompts without changing the named-root golds made the frozen score worse and did not teach whole-tree search. Keep 92/98. Remix the data. Do not ship the 82/98 checkpoint.