Shlex · local file-search model · 2026-09-18

Jeremiah v4: rows for three string rules, measured on a 312-case unseen board

What this run was for

Shlex is a desktop file-search pet. A 1.5-billion-parameter model on the person's Mac turns a plain request into one line that a sandboxed broker executes. Jeremiah v3 reached 80% on its held-out board, and reading its 39 failures showed that thirty of them were two string rules applied wrongly, not requests misunderstood: a provider name written as one word split in two (hugging face token), and the plural of a many-word topic missing or put on the wrong word (artworks lending). Four more were providers that are also ordinary words (resend) written as a phrase.

So this round wrote rows for exactly those shapes and nothing else: 560 accepted of 580 authored (192 compound-provider, 179 plural-topic, 60 common-word-provider, 43 document-noun and 30 refusal rows in train, 56 held out). Each row is a frozen intent, worded by Grok, passed through deterministic wording checks and a blind request-only review by a second Grok that never sees the intent; the blind reviewer agreed with every one of the 560 it saw, and the 20 rejections were wording-check failures. The mixture is the v3 training set unchanged (9,600 backbone + 2,400 v2 + 1,200 v3 rows) plus 496 v4 rows: 13,696 rows, 856 optimizer steps of 16, snapshots at 214, 428, 856, same recipe and same start point (jeremiah_v1) as v2 and v3.

How results are measured

Every number below is an executed test. The model's one-line answer is run through the real macOS broker on a fixture built from the case's expected plan, and the case passes only when the returned paths, or the refusal, match exactly. Generation is the app's own greedy decoding on Apple silicon. The trainer's token loss is logged and decides nothing. All four models were scored by the same script on the same boards on the same Mac.

The v3 and v4 development boards hold 96 and 56 unseen cases, small enough that one provider moves a family by two cases. So a fourth board was built through the same pipeline as the training rows, with vocabulary that no training, validation, development or acceptance split has ever contained: twelve compound providers, eight common-word providers, fifty-five topic sets, four repo names, six document nouns, two folders. 340 requests were authored, 326 passed the wording checks, 317 were accepted by the blind reviewer, 2 vendor-free requests were left off because they equal a training request and 3 duplicate wordings were left off: 312 cases, zero exact matches with the training set. Nothing on it will ever be trained on.

Unseen board

The intervals of the app model and v4 step 856 do not overlap. Every family except the plural family is at or above 93% for v4 step 856; scope wording, document nouns, secret spellings and refusals are perfect. Without the plural family the board reads 233/242 (96%) against 220/242 (91%).

v3 development board

109 cases from the v3 round, rebuilt from its accepted rows (hash-equal to the v3 freeze). 13 of them share their request text with a training row and count toward no gate. The huggingface and resend cases here are the two held-out providers the v4 rows could not touch.

v4 development board

This round's own held-out cases: providers, topic sets and nouns that appear in no v4 training row.

Operator canaries

Gates

Gates that can be judged on these boards: canaries 2/2; unseen v3 development at 90% (87 of 96) and each v3 family at 80%; zero false acceptances; zero unsafe executions. v4 step 856 clears the v3 unseen line at 88/96 (92%) and every v3 family except scope_synonyms (14/18 (78%), one case short, all four misses huggingface or resend). The v2 content-first gate (98/108), the targeted-development tolerance and the regression tolerance (within five of the app model's 1,246 of 1,330) are unevaluated for every v4 snapshot: those boards were not available on the Mac that scored this run. Nothing is nominated under the full gate set; the field-test release is an operator decision made on the boards above.

What still fails (raw model lines)

Of v4 step 856's 28 misses on the unseen board, 21 are the plural of a topic the model never saw. It learned the rule on its 55 training sets (the v4 development board moved from 5/17 to 10/17, the v3 topic sets from 19/23 to 22/23) and transfers it only partly: on new stems it drops the plural or edits the stem (mattress airting, lawn edger, loom heddels). Compound names it never saw pass 54/58; the misses are one name, MetaMask, split in two.

Update: the plural rule moved into the app

The first next step of this note, as first published, was to stop asking the model to recite a string rule. That is now done, in Shlex 0.7.1, and measured without training anything. Before a content search runs, the app reads the model's line and the person's request and adds to the any list only:

1. The spelling the person typed. A many-word phrase that is not in the request, but is one edited word away from a span that is, gains that span: lawn edger also gets lawn edging, flashling sealing gets flashing sealing.
2. Number. Every phrase gains the singular and the regular plural of its final word, by the rule the expected plans were built with, and the plural a person would write when English has an irregular one: kiln shelf, kiln shelfs and kiln shelves.

Nothing the model wrote is removed, so the plan that runs matches a superset of the documents the raw plan matches; every other field is untouched; a line that is already complete is returned byte for byte. Phrases the person quoted, and phrases with capitals or punctuation, are exact searches and are left alone. The app's history keeps the raw model line beside the plan that ran.

The measurement replays the cached lines of all four models, the same lines scored above, through the app's own binary and re-executes every line the rule changed on a fresh fixture. The rebuilt broker first reproduced all 479 recorded raw outcomes of v4 step 856, so the binary itself moves nothing.

The rule gives the model of the previous round 50 cases on the unseen board; this run's training gave it 45. The two add up: 304/312 (97%) for v4 step 856, and the plural family reads 69/70. The rule changed none of the 479 expected plans on these boards. In the training set it widens 769 expected plans of older, unquoted exact-phrase rows (renewal notice also gets renewal notices); all 769 were executed raw and widened and still pass.

The plural convention, decided. Number belongs to the app. The model names the topic and is no longer graded on reciting plurals; expected plans, boards and training rows stay as they are, mechanical forms such as tax advices included, because they match nothing and cost nothing; the app adds the real plural beside them. Scores are reported twice from here on: the raw line, and the line the app runs, which is what a person gets.

What this does not show. The rule was written while reading v4 step 856's failures on these same boards, so they are no longer unseen for the rule; the other three models are a partial check, not an independent one. The v2 development, regression and acceptance boards are still unscored, raw or as run.

None of the 16 is a plural. 6 are one more string rule; the rest are model behaviour, and are what the next round's rows are for.

Examples

Synthetic requests from the unseen board, raw model lines. Between the model in the app and v4 step 856: 45 cases fixed, 28 still failing, 0 newly failing.

The run

One RunPod Secure Cloud pod, RTX 6000 Ada, US-WA-1, created on the first request while the stock indicator read "Low" throughout (it read "Low" for the v3 run's seven refusals as well; the indicator does not predict creation). The Mac mini at home holds the API key, creates the pod, uploads a hashed packet without any board, watches the job, pulls every snapshot, verifies each hash, and only then deletes the pod; the GPU host never sees a credential. Training ran at 0.79 steps per second, the same as v3. Balance after the run: enough for ten more.

What changed since the v3 friction log: the controller is parametrised by contract, so v3 and v4 runs share one controller; the completion check follows the run's name; every step except training itself was rehearsed without a GPU. Still open: the 3 GB starting weights travel from home on every launch, and the pinned PyTorch stack is installed on every pod.

Next steps

1. Score the v2 boards and regression before anything is nominated. The v2 development board (403 cases), the 1,330-case regression board and the acceptance board were not scored for the v4 snapshots; they live on another Mac. The published gate set needs them: content-first 98/108, every family at 80%, regression within five cases of the model in the app, zero false acceptances. Score them twice, the raw line and the line the app runs. The app's rule can only add phrases, so it cannot lose a match there, but it has not been run on them. Read the regression false acceptances by hand, as the v3 note asked, before trusting a capability answer.
2. Fold a split compound name back together. With plurals gone this is the largest cluster left: six of sixteen misses. MetaMask still becomes meta mask in four unseen cases, HuggingFace in two v3 cases. It is the same kind of string rule as the plural one, and the same module could own it: when the person typed the joined form, the joined form is the only legal spelling. Otherwise a few hundred rows over camel-case names.
3. Keep common-word providers as providers. One unseen name (Front) and one v2 held-out name (Resend) are still written as a single phrase in all. The 60 v4 rows fixed every trained common-word name (39/40 unseen as well); a stored-credential contrast for more such names closes the rest. This one is model behaviour, not a string rule.
4. Small shapes worth thirty rows each. Request words kept inside the phrase (material on star chart, campsite booking material); a noun or "the project X" taken as the scope; a folder scope dropped; the type list dropped when newest ordering and a repo scope combine; a held-out document noun (spec) turned into a type. Each is one or two cases on every board.
5. A v5 round with no plural rows. The 179 plural rows of this round taught the training sets and half transferred. The app's rule gives the model of the previous round more on the unseen board than this whole run did. The next round should spend its rows on the clusters above and leave number to the app.
6. An independent check of the app's rule, and a human-authored acceptance set. The rule was written while reading this run's failures on these boards, so these boards are no longer unseen for the rule. Every request on every board except the two operator canaries is synthetic Grok wording. The field-test logs now flowing to the Mac mini are the first human requests at scale; they record the raw model line and the plan that ran, so a reviewed board built from them tests both.
7. Cheaper runs. Starting weights in private object storage instead of a home upload, a container image with the pinned PyTorch stack, and a balance check before creating a pod. This run cost 0.36 USD and got an RTX 6000 Ada on its first attempt; the v3 run paid 0.19 USD for four failed launches.

Identities

Environment: NVIDIA RTX 6000 Ada Generation, driver 570.124.06, Python 3.11.11, torch 2.11.0+cu128, transformers 5.15.1. Broker at scoring time: 865adb43c08bdbc5…; broker and rule at replay time: c75bb14a4c9466ac…, 016daa900b55dece….