In brief
How should document AI accuracy be measured?
Measure accuracy separately for each field and document type on a labelled test set drawn from real documents. Weight fields by business importance, set confidence thresholds per field, and track straight-through processing, correction rates and errors that reach downstream systems once live.
Vendors and project reports often summarise document extraction with a single number: the system is 95 percent accurate. That figure can hide exactly the errors that matter. If a system extracts twenty fields from an invoice and gets nineteen right, it is 95 percent accurate on that document. If the one wrong field is the total amount, the document still needs correction, and an unchecked error could lead to an incorrect payment.
Why averages mislead
- Easy fields, such as a vendor name printed in large type, inflate the average.
- Critical fields, such as amounts, tax values or account numbers, may be harder and less accurate.
- Accuracy varies widely by document layout, sender and scan quality.
- Document-level accuracy, where every required field is correct, is usually far lower than field averages suggest.
Build a proper test set
Select a random sample of real documents across senders, layouts, quality levels and time periods. Have trained reviewers label the correct value for each field, with a second review for a portion to check label quality. Keep this test set separate from any examples used to configure or train the system.
Report by field and document type
| Field | Importance | Accuracy measured | Notes |
|---|---|---|---|
| Invoice number | High | Per test set | Check for formats with prefixes or spaces |
| Invoice date | High | Per test set | Normalise formats before comparing |
| Total amount | Critical | Per test set | Validate against line items and tax |
| Tax identifiers | Critical | Per test set | Validate format and against master data |
| Line item descriptions | Medium | Per test set | Partial matches may be acceptable |
Fill in measured values from your own evaluation. The pattern of results, rather than a headline figure, shows where review effort and improvement work should focus.
Validation catches what extraction misses
Business rules add a second line of defence. Line items should sum to the subtotal, tax should match rates, dates should fall within plausible ranges, and supplier identifiers should exist in master data. A value that fails validation goes to review even if the model was confident.
Set thresholds per field
For each field, examine accuracy at different confidence levels in the test results. Choose the threshold above which accuracy meets the standard required for that field. Critical fields will need higher thresholds, which means more of them will be reviewed, and that is appropriate.
Design the review queue around uncertain fields
Reviewers should see the document image with uncertain fields highlighted and confident fields available for quick confirmation. Reviewing a single field in context takes seconds; re-keying a whole document does not. Every correction should be captured to improve the system.
Measure in production
| Measure | Purpose |
|---|---|
| Straight-through processing rate | Share of documents needing no human touch |
| Field correction rate | Where reviewers change values, by field and sender |
| Validation failure rate | Where business rules catch problems |
| Downstream error rate | Errors discovered after posting, the most important quality signal |
| Review time per document | Efficiency of the human review design |
Sources and further reading
About the author: Dr. AarzooCo-Founder and Chief Operating Officer, AARHIT Systems. Writes on delivery, quality assurance, data readiness, monitoring and human review. More articles by Dr. Aarzoo