Track 1 · Fine-tuning & Data Curation#6 Verifiable answersTiny Aya Global 3.35BLoRA · 4-bitFree T4 GPU

GeoDatum

A small model that calls a verified converter for Nigerian survey coordinates, and an honest look at where it still fails to ask. Tool calls went from 6 of 56 to 56 of 56 correct; asking when the datum or grid is missing did not improve (3 of 44 before, 0 of 44 after).

Before → after

First 100 test rows, same rows, same scorer and stop rule for both. “Before” is the untrained base model with the same system prompt.

Correct tool call

10.7% → 100%
Before6/56
After56/56
n = 56

End-to-end correct

7.1% → 37.5%
Before4/56
After21/56
n = 56 · about ±13 points

Asks when datum/grid missing

6.8% → 0.0%
Before3/44
After0/44
n = 44 · did not improve

Clean stop

24% → 100%
Before24/100
After100/100
n = 100

Main negative result

After training the model starts every reply with a tool call. It guesses UTM 31N when the grid is missing, fills in Minna when the datum is missing, and copies an unsupported UTM 33N straight from the user. Two causes are untested: the loss covered every token (assistant_only_loss=False), and 75 steps on 600 rows may be too little.

By scenario (first 100 test rows)

ScenarioScoredBeforeAftern
complete_insideend-to-end7.5%40.0%40
complete_outsideend-to-end6.2%31.2%16
missing_gridasks0.0%0.0%10
missing_datumasks7.7%0.0%13
missing_bothasks16.7%0.0%12
unsupported_zoneasks0.0%0.0%9

Why asking matters: same numbers, five grids

E 600000, N 892340 on the Minna datum lands in very different places:

GridLatitudeLongitude
UTM 31N8.072823.90677
UTM 32N8.072839.90687
West Belt12.050427.88961
Mid Belt12.070307.85110
East Belt12.031687.81597

Pipeline

1 · CONVERTER

pyproj/PROJ answer key and tool, five Minna grids

2 · GENERATOR

2,000 train · 600 test, six scenarios, no client data

3 · CURATORKIT

2,000 in, 2,000 out; 1,498 tool-call rows unchanged

4 · ALIGNTUNE

LoRA, 4-bit; 7.67M of 3.36B parameters (0.23%)

5 · SCORING

score.py and AuditKIT on the same 100 rows

What didn’t work

Limits and next steps

Limits

  • Synthetic, templated, in-distribution data
  • Minna datum only
  • “Inside Nigeria” is a bounding box
  • 600 of 2,000 rows, 100 test rows, one run
  • No real users yet

To test next

  • Loss on the reply only
  • Train longer on all 2,000 rows
  • Contrast pairs: same coordinates with and without the grid
  • Reward-based training with the converter as the reward
  • A check outside the model for missing inputs

Links

Reproduce

pip install pyproj curatorkit aligntune peft bitsandbytes auditkit
python geodatum_converter.py
python generate_dataset.py --n 2000 --seed 1 --out train.jsonl
python generate_dataset.py --n 600 --seed 2 --out test.jsonl --exclude train.jsonl
python curate.py train.jsonl output/clean_messages
python score.py --test test.jsonl --label after --adapter ./run1 --limit 100 --out after.json