- Whole-root versus child versus discourse: 36/36.
- Type versus filename versus same-token child: 36/36.
- Exact dotfile versus fragment versus content: 24/24.
- Zero repeated selected-root operands and zero scope escapes.
PorkiCoder Research · Bash SFT series RELEASE CANDIDATE · 233/240
233/240 on unseen human-like file search.
The only number that selects the model is a 240-item hard set written outside the training builder, with new tokens and fixtures, frozen before any generation on it. A targeted continuation scored 233. The previous champion scored 230 on the same Ada generations. The previous production model scored 25. That is the board.
Human-like hard set233/240Ada greedy generation, Darwin execution; same frozen 240
Prior champion230/240same Ada generator, prompts, and Darwin scorer
Old production model25/240run-last40-wd01 on the same 240
This is the eval we are optimizing for
Ordinary people do not type specification-shaped file-search prompts. The 240-item set is independently authored human-like language on new fixtures. It was frozen before any model saw it. That is the board. Older generator-matched sets and already-open private queries are not used to pick the ship candidate.
Release candidate: 233/240. A 2,180-row continuation from run-shlex-v31-human-transfer beat 230/240 on the same Ada generations and Darwin scorer. The previous production model scored 25/240. This is the Shlex ship checkpoint. Rollback is run-shlex-v31-human-transfer, then last40-wd01.
- Match/type+filename: 32/36 → 36/36.
- Unsupported pairs: 38/40 → 40/40. Zero false refusals, zero
$(…). - Exclusions: 17/20 → 20/20.
- Time slipped 47/48 → 43/48; one scope and one type/child row each dropped.
One board
Generation ran on an NVIDIA RTX 6000 Ada with greedy decoding, seed 42, and no semantic rewrite assistance. Commands were executed on macOS Bash 3.2. The prior champion was regenerated on the same Ada so 233 versus 230 is not a CUDA/MPS mix. Prompt-only SHA-256 de88de9e8685492eeffb3b8171c0622578ba6bba83bf13834f279f45a35595f0.
Family breakdown
| Family | r1 continuation | Prior champion | Readout |
|---|---|---|---|
| Whole root versus child versus discourse | 35/36 | 36/36 | One stdout miss. |
| Type versus filename versus same-token child | 35/36 | 36/36 | One stdout miss. |
| Exact dotfile versus fragment versus content | 24/24 | 24/24 | Held. |
| Time conversion and newest | 43/48 | 47/48 | Four extra recency mismatches. |
| Filename / content / topic plus type | 36/36 | 32/36 | PowerPoint and false image-content refusals closed. |
| Unsupported / representable pairs | 40/40 | 38/40 | Wrong slug and $(…) gone. |
| Exclusions and retained skills | 20/20 | 17/20 | Prune/or-precedence closed. |
| Total | 233/240 | 230/240 | Net +3 on the ship board. |
All 240 r1 outputs were one physical line, policy-valid, and left the fixtures unchanged. Authors split 113/120 and 120/120. Zero repeated selected-root operands, zero path escapes, zero false unsupported outcomes.
How far we have come—and what the numbers stopped meaning
The early notes asked whether a small model could learn a narrow Bash contract. Note 05 asks whether a larger continuation transfers to independently authored human-like requests. That is now the only public selection number.
| Public checkpoint | What changed | Headline result | What it established |
|---|---|---|---|
| Note 01 | 137-row first SFT | 14.0% exact | A tiny run could move a held-out imitation score. |
| Note 02 | 3,000 executed examples | 89/98 | The narrow contract was learnable under a fair prompt. |
| Note 03 | File-search specialization | 92/98 | The best frozen synthetic checkpoint became the incumbent. |
| Note 04 | Shorter live-style continuation | 82/98 · 2/27 | More targeted training could regress both the old and new boards. |
| Human canary report | Faithful conversational SFT | 79/80 · 4/13 | A near-perfect synthetic score still failed the human release gate. |
| Note 05 | 10,202-row champion, then 2,180-row targeted continuation | 233/240 | Unseen human-like transfer versus 230/240 for the prior champion and 25/240 for the previous production model. |
The lesson so far: older frozen Bash sets answer a different question. For shipping Shlex, the human-like 240 is the board. Averaging it with generator-matched diagnostics would hide the thing we actually care about.
8,500 new records, with the author written on every row
The finished corpus contains 7,800 new training rows, 700 new validation rows, 2,402 replay rows from the first frozen file-search set, and 297 replay validation rows. New language came from one deterministic renderer and three local model families.
Every record accepted, with zero missing and zero rejected at canonical intake.
Training plus validation rows. Two independent builds produced identical bytes.
| Final author | New records | Share (rounded) | Role |
|---|---|---|---|
| Qwen2.5-7B-Instruct | 3,362 | 39.6% | Primary local fallback plus bounded residuals. |
| Phi-4-mini-instruct | 2,465 | 29.0% | Independent-generator allocation plus a small residual. |
| Mistral-7B-Instruct-v0.3 | 1,698 | 20.0% | Third-generator allocation plus bounded residuals. |
| Deterministic renderer | 975 | 11.5% | Template-controlled intents and commands. |
What happened to Grok
The first Grok attempt went through PorkiConsult, which called an Ada-path helper at /root/.grok/bin/grok. That is the wrong binary on this Mac. Zero Grok-written rows entered the 10,202-row corpus; Qwen, Phi, Mistral, and the deterministic renderer finished it. The next continuation authors targeted utterances with the signed-in local CLI at ~/.grok/bin/grok.
The most useful results happened before training
The preparation job failed twice for good reasons. Each failure prevented a cleaner-looking but less trustworthy experiment.
437 extra attempts isolated
The recursive scan found partial provider attempts and diagnostics. They were preserved in a private quarantine outside canonical intake—not deleted and not silently merged.
Six collision groups repaired
Punctuation normalization exposed train/validation overlap. Seven synthetic validation rows were backed up and regenerated.
Zero overlap at freeze
The combined replay-plus-new audit ended with zero normalized cross-split collisions, then reproduced byte for byte on a second build.
That is the operational gain in note 05: failed attempts stay visible, author provenance survives fallback, split leakage is checked after normalization, and the final corpus is frozen by hash before training.
One fixed recipe, 61 seconds
The last run was a targeted continuation from the 230/240 champion, not a second 10,202-row SFT. Response-only full SFT on one NVIDIA RTX 6000 Ada Generation GPU. Hyperparameters stayed fixed. Kimi K3 signed off on lr 5e-6 before the droplet was created.
| Field | Value | Interpretation |
|---|---|---|
| Init | run-shlex-v31-human-transfer | The 230/240 champion, not stock and not last40. |
| Train / val rows | 2,180 / 78 | 660 new miss-family rows plus 1,520 replay of families already at 100%. |
| Epochs / optimizer steps | 1 / 137 | One pass; effective batch 16. |
| Learning rate / schedule | 5e-6 / cosine | Lower than the 1e-5 continuations that previously forgot the contract. |
| Weight decay / max length | 0.1 / 1,024 | Fixed before the run. |
| Train loss / val loss | 0.002779 / 0.007823 | Optimization diagnostics, not the product score. |
| Runtime / peak GPU memory | 60.64 s / 16.77 GiB | The 48 GB Ada had substantial headroom. |
What the synthetic passes and misses looked like
These are synthetic items from the 240-item board. They are not private user queries. The two miss rows are the prior champion's remaining error classes; r1 closed both families.
| Result | Synthetic request | Candidate output | Expected behavior |
|---|---|---|---|
| Pass | i need html files here | find . -type f \( -iname '*.html' -o -iname '*.htm' \) -print | sort | Whole-tree language keeps the selected root as . |
| Pass | i'm looking for html files inside the annex folder | find annex -type f \( -iname '*.html' -o -iname '*.htm' \) -print | sort | Named child stays a child. |
| Pass | i need files with html in the filename in the selected folder | find . -type f -iname '*html*' -print | sort | Filename fragment, not HTML file type. |
| False refusal | images containing flint-anvil-c49037 in this directory | unsupported:needs-image-content | Text inside image files is a representable search. |
| Wrong type | slide decks with nimbus-anvil-9c6676 in the filename in this directory | find . -type f -iname '*nimbus-anvil-9c6676*' \( -iname '*.md' -o -iname '*.markdown' \) -print | sort | PowerPoint extensions, not Markdown. |
The public packet contains no private query wording, personal path, raw prediction bundle, or fixture.
The continuation shipped on this board
- Match/type+filename closed: 32/36 → 36/36.
- Unsupported pairs closed: 38/40 → 40/40.
- Exclusions closed: 17/20 → 20/20.
- Net +3 versus the prior champion on the same 240.
- Time conversion 47/48 → 43/48.
- One whole-root/child miss (35/36).
- One type/filename/child miss (35/36).
- Formal family floors still fail on time. The operator board is the 240 total.
Release candidate: run-shlex-v31r1 at 233/240. Rollback run-shlex-v31-human-transfer (230/240), then last40-wd01 (25/240). Product wiring is a later session. No further training run is planned on this board.
The GPU left only after the weights arrived
The r1 model.safetensors is 3,087,467,144 bytes. It was pulled locally and hashed before the Ada was deleted. The tensor file is not published. The prior champion remains on disk as rollback.
Ship SHA-256
8ccf00779bf93e16981c6f0272197a768165c6a21a6c8b5e4f81a9b1f9371ef0
Sanitized evidence
The public evidence packet contains aggregate corpus and evaluation metrics, hashes, family counts, and five synthetic examples. It excludes training and validation JSONL, partial provider output, private prompts and paths, raw evaluations, fixtures, credentials, and all weight tensors.
Bottom line
233/240 is the board. The prior champion scored 230/240 on the same Ada generations. The previous production model scored 25/240. run-shlex-v31r1 is the release candidate.