What Happened When I Measured My Vision Pipeline

I have a pipeline that reads photographed Bhatkhande notation and returns structured, editable notation. I wrote up how it works — font-reference grounding, raga and taal conditioning, per-section fan-out, prompt caching.

Every time anyone asked how accurate it was, I said "roughly 90% on printed pages in manual testing." My own issue tracker says 94%. Neither number came from a measurement. They came from watching good scans and feeling pleased.

Then I built the eval, and the measurement turned out to be the least interesting thing it produced.

Ground truth was already in the database

The hard part of an eval like this is meant to be labelling. Twenty pages of notation, typed by hand, is a day's work — and if you label by correcting model output you anchor yourself into waving through plausible errors.

I didn't need to. The compositions in my database are already labelled data. Someone typed each one, and the app renders them. So: render a composition, screenshot it, scan the screenshot, diff against the stored notation. 24 fixtures, 2,978 beats, zero labelling, and no anchoring bias because nothing was corrected against a prediction.

The section hints come free too, computed the same way the app computes them — rows from the taal's beats-per-row, and leading empty beats from the notation itself.

One honest limit: a screenshot of my own renderer isn't a photograph of a book. Even glyphs, perfect contrast, no page curl, no camera. Every number below is an upper bound, and a good regression harness. It is not "accuracy on printed pages".

Thinking moves the number as much as the model does

Here is what the corrected sweep found, three pages, same images, model verified from the function log each time:

Two things move this number, and I went in expecting only one of them to matter.

Thinking moves it a lot. Opus at effort: low scores in the seventies. The same model at its default effort scores 93.8% — call it eighteen points from the effort setting alone, on one model. The accuracy is substantially in the thinking tokens. Reading a notation grid is apparently not perception, it's deliberation.

But the model matters at least as much, and I have to say so, because my own table says so. Hold effort fixed at low and Opus scores 75.7% where Sonnet scores 49.8% — twenty-six points, from nothing but the choice of model. That gap is larger than the effort gap within Opus. I started this write-up planning to claim effort dominates model choice; the numbers below do not support it, and the honest version is that on this task the two are comparable and neither is safe to guess at.

The Haiku row wants a caveat too. Haiku 4.5 rejects the effort parameter, which is why the run was configured without extended thinking — but that is a limitation of effort, not of the model: Haiku 4.5 does support extended thinking through the thinking parameter with an explicit token budget. So 25.2% is Haiku with thinking switched off, which is not the same as Haiku's ceiling. I have not measured the latter, and until I do I cannot actually tell you whether the cheap path exists. What I can say is that the cheap path is not simply "swap in the small model and keep everything else."

There is a catch, and it is the reason effort: low was pinned in the first place: at default effort, Opus and Sonnet timed out on two of three pages, spending the whole output budget on reasoning before emitting JSON. The production setting was chosen to stop that, and it quietly cost about eighteen points of accuracy against Opus's median low run.

The exception: Sonnet is worse and slower

I expected Sonnet to be the sensible middle — cheaper than Opus, more capable than Haiku, the obvious place to land. It is neither.

At effort: low, on the same three pages: 49.8%, against Opus's 75.7% at the same setting. Twenty-six points worse. And the median call took 89 seconds against Opus's 45.

Raising the effort does not rescue it. At medium it reached 64.1% — still below Opus at low — and timed out on two of the three pages, taking 114 seconds on the one that finished. At default effort it timed out on two of three as well. There is no setting at which Sonnet is both reliable and good here: low finishes and scores badly, medium and above score better and do not finish.

So the ordering is not price and it is not size. Whatever makes this task work is distributed across these models in a way I could not have predicted from the outside, which is the whole argument for measuring instead of reasoning about it. Left to my own judgement I would have shipped Sonnet, on the grounds that it was a safe middle.

Where the money actually goes

The function logs token usage. Three pages on Sonnet at effort: low:

9 calls:  input 4,055   output 50,036   cache-write 31,953   cache-read 53,571

Input is four thousand tokens. Output is fifty thousand. About 16,700 output tokens per page, at the lowest effort setting that still thinks.

That reframes the cost question entirely. The font reference sheet and the page image — the things I assumed were expensive, and had spent an afternoon planning to optimise by tiling and downscaling — are a rounding error. The cost is reasoning tokens. And the cache is already doing its job on the input side: 53,571 read against 31,953 written.

I nearly spent a day making the cheap half cheaper.

Fable 5.1 is the one configuration that got both: 90.7% without timing out, at half Opus's latency.

Three runs, same config, same page: 59.4%, 59.4%, 90.6%

Because temperature cannot be pinned, I have three independent samples of one configuration. On one page they scored 59.4%, 59.4% and 90.6%.

That's a 31-point spread with nothing changed. It means any single-run accuracy claim about this pipeline is noise — including the 94% I had written in my own issue tracker, and the "roughly 90%" I had been repeating.

The failures are alignment, not glyphs

Diffing a bad run against a good one on the same page was the most useful ten minutes of the whole exercise:

exp "N"   got "p"      exp "-"   got "D"
exp "su"  got "m"      exp "p"   got "-"

Those aren't misread marks. Those are the wrong cells — a note where a rest is, a rest where a note is, values shifted. The good run's errors were a strict subset of the bad run's, and the bad run had lost grid alignment partway through.

Beat counts were perfect on every page, every model, every run — 128/128, 160/160, 144/144. The section hints guarantee the right number of beats, so the drift hides inside a correctly-shaped grid. A structurally valid answer that is substantively wrong is the hardest kind to notice.

I had assumed the dominant error was octave dots, because notes-only accuracy runs consistently 6–7 points above octave-level. That gap is real and it's worth fixing. But it's second-order next to a section occasionally losing its place.

One thing I was wrong about twice

I read cache write: 9425, cache read: 0 in the logs and concluded the prompt cache was never being read — that the warm-up call was writing a prefix nothing then read, which would make caching strictly worse than not caching.

I was reading two interleaved lines from different scans. The clean sequence is exactly right:

cache write: 9425, cache read: 0      warm-up
cache write: 0,    cache read: 9425   section 1
cache write: 0,    cache read: 9425   section 2

One write, N reads, as designed. Worth saying because I nearly "fixed" something that worked.

What I still cannot tell you

Three pages is a small sample, and the page-to-page spread is wider than most of the differences I just reported. The images are renders rather than photographs. And I have no measurement at all for handwriting, which is the case people actually ask about.

But "roughly 90%" is now retired. The defensible claim is narrower and more useful: on rendered pages, with the grid supplied, Fable 5.1 at default effort gets about 90% of beats exactly right including octave and ornament, and the same pipeline on Haiku 4.5 with thinking switched off gets 25%.