98% field accuracy. Is the document right?

02Evaluate the business task at field, document and workflow levels, using representative documents and independently checked answers.

03Separate language, layout and image quality.

04A model that reads a paragraph well can still attach a quantity to the wrong line item or confuse an invoice date with a due date.

05The decision is whether a defined workflow can be assisted safely at an acceptable total effort.

06It is not which model leads a general leaderboard.

07This explanation applies to bilingual operating documents in several markets; it does not claim one Arabic benchmark represents Saudi and UAE paperwork equally.

02Its breadth demonstrates why “Arabic accuracy” needs task definition.

03A benchmark score is conditional on its dataset and evaluated model versions.

04Heakl et al., KITAB-Bench, Findings of ACL 2025, pages 22006–22024, checked 16 September 2026 (source 2).

05DocVQA introduced approximately 50,000 questions over more than 12,000 document images and emphasised questions requiring document structure.

06It is useful precedent for evaluating answers in document context, but it is not a country-specific invoice compliance test.

07Mathew, Karatzas and Jawahar, DocVQA, July 2020 (source 3).

08Arabic document work also involves cursive text and right-to-left reading within potentially mixed-direction layouts.

09The KITAB paper treats these as evaluation challenges.

10Do not turn its aggregate comparisons into a vendor performance promise for photographed purchase orders.

11KITAB-Bench paper, 2025 (source 1).

12For experimental discipline, the NIST AI RMF highlights context, measurement and risk management rather than a single universal score.

13This explanation's proposed test protocol is dotSuper's operational interpretation, not NIST certification.

14NIST AI RMF 1.0, January 2023, checked 16 September 2026 (source 4).

02For a quality report, the critical fields may be drawing revision, measured dimension, tolerance and pass/fail evidence.

03The business owner defines the consequences of each error before the test starts.

04Measure character error rate only for transcription tasks.

05It counts character substitutions, deletions and insertions relative to reference characters.

06It does not measure whether an extracted bank account belongs to the correct supplier.

07A semantically plausible paraphrase may be acceptable for a summary but unacceptable for an identifier.

08Use exact comparison for identifiers where appropriate.

09For names and addresses, document any normalisation rules in advance.

10Preserve the raw text alongside the normalised value.

11Do not silently erase leading zeros, punctuation or script differences merely to raise a score.

12For numeric amounts, distinguish the value from its currency and sign.

13For tables, score row association as well as field content.

14Ten correct quantities attached to the wrong products are ten wrong business records.

15Include subtotal, discount and credit-note cases.

16Require the system to return missing or uncertain when the document does not support an answer.

02Stratify by Arabic, English and mixed content; native PDFs and scans; clear and degraded images; known and new supplier templates; single and multiple pages.

03Document whose production mix the test represents and which cases are deliberately overrepresented for safety testing.

04Keep supplier templates and near-duplicate documents grouped when splitting development and test data.

05A random page split can leak the same form into both sets.

06Freeze the final test set and model configuration before reporting results.

07Record extraction schema, prompt, model version, preprocessing and any external lookup used.

08Have a second reviewer check the reference answers for critical fields.

09Resolve disagreement explicitly.

10An ambiguous source should not be counted as a clear model error or a clear success.

11Keep an adjudication log with the document locator and reason.

12Include adversarial and operational cases: a page that instructs the system to change its rules, an illegible amount, a missing page, a duplicate invoice, conflicting totals and a new stamp obscuring text.

13These are test inputs.

14They do not authorise changes to the system's operating instructions.

02Assume 200 documents with five evaluated fields each, giving 1,000 field decisions.

03Twenty field errors are spread across 20 different documents.

04Field accuracy is 980/1,000, or 98%.

05Fully correct documents are 180/200, or 90%.

06The table is constructed so that the 2, 12 and 6 incomplete documents account for the 20 total errors, one per affected document.

07It does not report a real model's language gap.

08The point is that a weighted overall score can hide a weak subgroup and a small denominator.

09Suppose each document saves three minutes of entry but requires one minute of routine review.

10Gross released effort is 200 × 2 = 400 minutes before exception correction, test maintenance and integration.

11If exceptions add five minutes on each of the 20 affected documents, net released effort becomes 300 minutes, or five hours.

12This remains a capacity estimate, not a realised cash saving.

02Twenty mixed documents are too few to make a precise general claim about all mixed-language invoices.

03Expand the sample where the business risk or uncertainty matters, rather than evenly adding easy documents to improve the overall average.

04For an independent Bernoulli-error approximation, observing zero errors in 100 cases still allows an upper one-sided 95% error bound of about 3%.

05This follows from solving (1-p)^100 = 0.05.

06Real documents may be correlated, making that simple bound optimistic.

07Zero observed errors is not a guarantee of zero production errors.

08Acceptance should combine field correctness, exception detection, reviewer workload and downstream error severity.

09Define unacceptable errors and the required escalation in advance.

10A system may be acceptable for draft indexing but unsuitable for unattended accounting entries using the same underlying model.

02Each source supports only the scope stated beside the claim.

02Record both complete-document results and reviewer effort.

02Conversely, unfamiliar layouts can benefit from broader document understanding.

03Evaluate both against the same business task rather than assume novelty determines suitability.

04Human review also makes errors and can become perfunctory.

05Measure whether reviewers actually catch seeded mistakes and whether the interface shows sufficient source context.

06A workflow that routes everything to a human may be safe in one sense but economically unhelpful or operationally overloaded.

07Benchmark versions age.

08No current model ranking or Arabic-English production accuracy is asserted here.

09The dataset's licence and any underlying document restrictions must be checked separately before commercial testing.

10A research paper being freely readable does not grant rights to its source images.