A small model that calls a verified converter for Nigerian survey coordinates, and an honest look at where it still fails to ask. Tool calls went from 6 of 56 to 56 of 56 correct; asking when the datum or grid is missing did not improve (3 of 44 before, 0 of 44 after).
First 100 test rows, same rows, same scorer and stop rule for both. “Before” is the untrained base model with the same system prompt.
After training the model starts every reply with a tool call. It guesses UTM 31N when the grid is missing, fills in Minna when the datum is missing, and copies an unsupported UTM 33N straight from the user. Two causes are untested: the loss covered every token (assistant_only_loss=False), and 75 steps on 600 rows may be too little.
| Scenario | Scored | Before | After | n |
|---|---|---|---|---|
| complete_inside | end-to-end | 7.5% | 40.0% | 40 |
| complete_outside | end-to-end | 6.2% | 31.2% | 16 |
| missing_grid | asks | 0.0% | 0.0% | 10 |
| missing_datum | asks | 7.7% | 0.0% | 13 |
| missing_both | asks | 16.7% | 0.0% | 12 |
| unsupported_zone | asks | 0.0% | 0.0% | 9 |
E 600000, N 892340 on the Minna datum lands in very different places:
| Grid | Latitude | Longitude |
|---|---|---|
| UTM 31N | 8.07282 | 3.90677 |
| UTM 32N | 8.07283 | 9.90687 |
| West Belt | 12.05042 | 7.88961 |
| Mid Belt | 12.07030 | 7.85110 |
| East Belt | 12.03168 | 7.81597 |
pyproj/PROJ answer key and tool, five Minna grids
2,000 train · 600 test, six scenarios, no client data
2,000 in, 2,000 out; 1,498 tool-call rows unchanged
LoRA, 4-bit; 7.67M of 3.36B parameters (0.23%)
score.py and AuditKIT on the same 100 rows
<|END_RESPONSE|> and began a new turn, and the scorer didn’t stop on that token.pip install pyproj curatorkit aligntune peft bitsandbytes auditkit python geodatum_converter.py python generate_dataset.py --n 2000 --seed 1 --out train.jsonl python generate_dataset.py --n 600 --seed 2 --out test.jsonl --exclude train.jsonl python curate.py train.jsonl output/clean_messages python score.py --test test.jsonl --label after --adapter ./run1 --limit 100 --out after.json