# Qwen2.5-Coder-1.5B-Instruct: base vs full SFT

- val rows: 500
- base: `Qwen/Qwen2.5-Coder-1.5B-Instruct`
- sft: 137-row 1-epoch response-only checkpoint of `Qwen/Qwen2.5-Coder-1.5B-Instruct`
- device: `cuda`, dtype: `torch.bfloat16`
- decoding: greedy, max_new_tokens=256

## Overall

| metric | base | sft | delta |
|---|---:|---:|---:|
| n | 500 | 500 | |
| teacher-forced loss | 1.5155 | 1.2693 | -0.2463
| teacher-forced token acc |  69.7% |  76.3% | +6.6%
| has ```bash fence |  99.8% | 100.0% | +0.2%
| SFT-style prefix |   0.0% | 100.0% | +100.0%
| exact bash match |  12.2% |  14.0% | +1.8%
| first-line match |  12.4% |  14.0% | +1.6%
| command-stem match |  50.4% |  56.8% | +6.4%
| first-tool match |  86.4% |  91.6% | +5.2%
| extracted-bash token F1 |  41.5% |  43.8% | +2.4%
| full-text token F1 |  41.4% |  66.3% | +24.9%
| empty completion |   0.0% |   0.0% | +0.0%
## Kind: task_solution

| metric | base | sft | delta |
|---|---:|---:|---:|
| n | 500 | 500 | |
| teacher-forced loss | 1.5155 | 1.2693 | -0.2463
| teacher-forced token acc |  69.7% |  76.3% | +6.6%
| has ```bash fence |  99.8% | 100.0% | +0.2%
| SFT-style prefix |   0.0% | 100.0% | +100.0%
| exact bash match |  12.2% |  14.0% | +1.8%
| first-line match |  12.4% |  14.0% | +1.6%
| command-stem match |  50.4% |  56.8% | +6.4%
| first-tool match |  86.4% |  91.6% | +5.2%
| extracted-bash token F1 |  41.5% |  43.8% | +2.4%
| full-text token F1 |  41.4% |  66.3% | +24.9%
| empty completion |   0.0% |   0.0% | +0.0%
## Cloud rows (aws/doctl)

| metric | base | sft | delta |
|---|---:|---:|---:|
| n | 9 | 9 | |
| teacher-forced loss | 1.7775 | 1.4818 | -0.2957
| teacher-forced token acc |  65.1% |  71.4% | +6.3%
| has ```bash fence | 100.0% | 100.0% | +0.0%
| SFT-style prefix |   0.0% | 100.0% | +100.0%
| exact bash match |   0.0% |   0.0% | +0.0%
| first-line match |   0.0% |   0.0% | +0.0%
| command-stem match |  55.6% |  55.6% | +0.0%
| first-tool match |  55.6% |  55.6% | +0.0%
| extracted-bash token F1 |  18.3% |  19.8% | +1.5%
| full-text token F1 |  24.7% |  49.5% | +24.8%
| empty completion |   0.0% |   0.0% | +0.0%
## Non-cloud rows

| metric | base | sft | delta |
|---|---:|---:|---:|
| n | 491 | 491 | |
| teacher-forced loss | 1.5107 | 1.2654 | -0.2454
| teacher-forced token acc |  69.8% |  76.4% | +6.6%
| has ```bash fence |  99.8% | 100.0% | +0.2%
| SFT-style prefix |   0.0% | 100.0% | +100.0%
| exact bash match |  12.4% |  14.3% | +1.8%
| first-line match |  12.6% |  14.3% | +1.6%
| command-stem match |  50.3% |  56.8% | +6.5%
| first-tool match |  87.0% |  92.3% | +5.3%
| extracted-bash token F1 |  41.9% |  44.3% | +2.4%
| full-text token F1 |  41.7% |  66.6% | +24.9%
| empty completion |   0.0% |   0.0% | +0.0%