One document problem. Three AI builds.

Somewhere in your business, a person is reading a PDF and typing what it says into a system that already has a field for it.

Supplier invoices. Purchase orders. Work orders. Delivery notes. Every vendor formats them differently, half arrive as scans, and the ones that arrive as clean digital files still put the totals in a different place each time. The work is dull, it's expensive, and it's the single most common thing companies ask us to automate.

It's also the job where the "which AI should we use?" question has the least obvious answer. So rather than argue about it, we built the same extraction feature three different ways and compared them on the things that decide a project: accuracy, speed, what it costs to run, and how long it takes to get live.

One note on why this comparison keeps mattering even as models change. Model prices fall and capabilities rise, but the ratio between a frontier model and a small one for a narrow, repetitive task has stayed roughly an order of magnitude for years. The specific numbers date quickly. The shape of the decision doesn't.

Same input, same twelve fields out, same test set of documents from a spread of vendors including scans and photos taken on phones.

Accuracy here is measured per field, not per document, for reasons covered further down. Run cost is expressed relative to Build A, because ratios survive price changes and absolute figures don't.

Read the table as a shape rather than a scoreboard. Build A wins on time-to-live and resilience. Build C wins on everything that matters once you're running at volume, provided the documents keep looking roughly the same. Build B is the one most teams should be on, because it captures most of the cost saving without the labeling project.

The accuracy result is worth sitting with. The fine-tuned small model beat the frontier model not because it's a better model — it plainly isn't — but because it had seen two thousand examples of exactly this task, labeled by people who knew what the right answer was. Data beat model size on a narrow task. That's not a surprising result to anyone who has done it, and it's consistently surprising to everyone who hasn't.

How we'd actually choose

• Under ~5,000 documents a month — Build A. The run cost barely registers and you'll be live next week.
• 5,000 to 50,000, layouts fairly stable — Build B. This is the sweet spot for most businesses.
• Above ~50,000, or tight latency, or data must stay on-premise — Build C. The labeling cost pays back quickly at this volume.
• Layouts change constantly, or vendors churn — stay on A or B regardless of volume. Fine-tuning rewards stability and punishes change.

And one that applies to all three: route low-confidence fields to a human. Every build above has a confidence signal available, and a review queue for the bottom few percent delivers more real-world accuracy than any amount of model upgrading.

The part that does most of the work

Across all three builds, the single highest-leverage piece of code was the validator, not the prompt. Declaring the output shape and checking it against business rules turns an unreliable component into a reliable system — and it's the same handful of lines regardless of which model sits underneath.

# Don't trust the model to be correct. Make the system prove it.
from datetime import datetime

REQUIRED = ["invoice_no", "vendor", "date", "currency", "total", "lines"]

def validate(doc, allowed_currencies):
    errors = []

    for field in REQUIRED:
        if doc.get(field) in (None, "", []):
            errors.append(f"missing: {field}")

    try:
        datetime.strptime(doc["date"], "%Y-%m-%d")
    except (KeyError, ValueError):
        errors.append("date not in YYYY-MM-DD")

    if doc.get("currency") not in allowed_currencies:
        errors.append(f"unexpected currency: {doc.get('currency')}")

    # The check that catches the most real errors: does the maths work?
    line_total = sum(l["qty"] * l["unit_price"] for l in doc.get("lines", []))
    if abs(line_total - doc.get("total", 0)) > 0.02:
        errors.append(f"lines sum to {line_total}, total says {doc.get('total')}")

    return errors

# Failed once -> retry with the errors fed back in.
# Failed twice -> human review queue. Never silently accept.

Our figures don't matter; yours do. Enter your volumes below. Everything is calculated in your browser and nothing is sent anywhere.

Most extraction projects are evaluated on the wrong number. "The model is 94% accurate" usually means 94% of documents were perfect, which tells you almost nothing useful — a document with eleven correct fields and one wrong total counts as a failure, and so does one with twelve wrong fields.

Measure per field, and weight the fields by what it costs you to get them wrong. A misread invoice number is an annoyance someone fixes in a minute. A misread total is money leaving the business. Those should never share a score.

• Accuracy per field, not per document — so you can see which field is actually failing and fix that one.
• Weighted by cost of error — the total, the currency and the vendor matter more than the description line.
• Separate "wrong" from "didn't answer" — a field the system declined to guess is a success, because it went to review instead of into your accounts.
• Track the review rate too — a build that's perfectly accurate by sending 40% to humans hasn't automated anything.

How to run this comparison yourself in a week

• Day 1 — collect 200 real documents spanning your messiest vendors, and write down the correct answer for each field by hand.
• Day 2 — build A. A prompt, a schema, a loop over your 200 documents.
• Day 3 — score it per field. You now have a baseline that no opinion can argue with.
• Day 4–5 — swap in a smaller model, add validation and retry, and score again on the same 200.
• End of week — you'll know whether fine-tuning is worth scoping, and you'll have the evaluation set that every future change gets measured against.

That last point is the one worth keeping. The models in this comparison will all be superseded. The 200 labeled documents won't be — they'll still be telling you, two years from now, whether the newest thing is actually better for your business or just newer.