The benchmark, and what it can and cannot tell you
Every converter in this market publishes an accuracy figure from a private test set that nobody else can run. This page is the opposite: the statements, the ground truth, the script that scores them and every result, so that anyone can check the figures and reproduce them.
Read this first. Every statement here is synthetic, generated by a script we wrote, and the engine was built by the same people. A full score on a corpus like that mostly shows the engine does what we thought to test. It says nothing about how any real bank's PDF converts. Until real statements have been through it — with the owners' consent and redacted — there are no per-bank accuracy figures on this page, and the bank-shaped statements below are layouts, not banks.
The method
Each statement is generated with known ground truth: every row's date, description, amount and running balance. The converter's engine reads the statement's positioned text with no hints — no date order, no bank profile — the way the browser reads a file it has never seen. Every field is then compared with ground truth, exactly: an amount one penny out is wrong, and a description with a missing word is wrong.
Each statement also has an expected outcome. Most should reconcile. A statement with no running balance should have every row reported as not checkable, and never as verified. A statement that contradicts itself should be flagged at the contradiction and nowhere else. A scan should be refused, not guessed at.
Results
38 of 38 statements reached their expected outcome. Of 897 rows: 896 amounts, 877 of 877 printed balances, 897 dates and 897 descriptions were exactly right.
The amount that did not match was on the statement built to contradict itself: it prints one figure where its own running balance moved by another. The engine reads what is printed, and flags the row — which is the right answer, and the reason that amount does not match ground truth.
| Statement | Expected outcome | Rows | Amounts | Balances | Dates | Descriptions |
|---|---|---|---|---|---|---|
| ruled Ruled table, money in / money out / balance | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| unruled The same layout with no table lines | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| wrapped References wrapped onto a second line | Met Reconciled | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| multipage Five pages with brought- and carried-forward rows | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| drcr One amount column marked DR / CR | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| parens Money out in parentheses | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| eu European number format, 1.234,56 | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| mdy US month-first dates, deposits before withdrawals | Met Reconciled | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| reversed_cols Money out before money in | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| no_headers No column headers: direction settled by the arithmetic | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| india Lakh grouping, 1,23,456.78, password-protected | Met Reconciled | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 |
| long 240 rows over ten pages | Met Reconciled | 240/240 | 240/240 | 240/240 | 240/240 | 240/240 |
| batch_q1 January to April, first of three overlapping statements | Met Reconciled | 80/80 | 80/80 | 80/80 | 80/80 | 80/80 |
| batch_q2 April to July, overlapping both neighbours | Met Reconciled | 80/80 | 80/80 | 80/80 | 80/80 | 80/80 |
| batch_q3 July to December, password-protected | Met Reconciled | 120/120 | 120/120 | 120/120 | 120/120 | 120/120 |
| split_words Every word its own text item, day-month dates split in two | Met Reconciled | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| sample_chase Shaped like the Chase layout recorded on this site: MM/DD, no year | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| sample_barclays Shaped like the Barclays layout recorded on this site: DD Mon, no year | Met Reconciled | 10/10 | 10/10 | 10/10 | 10/10 | 10/10 |
| sample_hdfc Shaped like the HDFC Bank layout recorded on this site: DD/MM/YY, lakh | Met Reconciled | 8/8 | 8/8 | 8/8 | 8/8 | 8/8 |
| claim_opening_each_page Opening balance reprinted on every page (a claim on the Chase page) | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| claim_fee_table A fee table after the closing balance (a claim on the Chase page) | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| claim_bilingual English and French header lines (a claim on the Canada page) | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| overdrawn_minus Overdrawn account, negative balance printed -54.52 | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| overdrawn_od Overdrawn account, negative balance printed 54.52 OD | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| overdrawn_dr Overdrawn account, negative balance printed 54.52 DR | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| overdrawn_parens Overdrawn account, negative balance printed (54.52) | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| year_end Across the new year, day-month dates with no year | Met Reconciled | 7/7 | 7/7 | 7/7 | 7/7 | 7/7 |
| date_once_per_day The date printed only on the first transaction of each day | Met Reconciled | 7/7 | 7/7 | 7/7 | 7/7 | 7/7 |
| currency_symbol A currency symbol on every amount and balance | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| balance_end_of_day The balance printed only once a day | Met Every day adds up; 2 single-row days verified, 5 rows checked only as part of a day | 7/7 | 7/7 | 4/4 | 7/7 | 7/7 |
| newest_first Transactions listed newest first | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| amount_on_last_line Amount and balance on the last line of a wrapped entry | Met Reconciled | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| two_accounts Two accounts in one PDF, each with its own balances | Met Reconciled | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| mdy_early_days Month-first dates all on or before the 12th; only the period decides | Met Reconciled | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 |
| nobalance No running balance column | Met Every row reported as not checkable, and said so | 12/12 | 12/12 | none printed | 12/12 | 12/12 |
| claim_dual_dates Credit card: transaction and posting dates, no running balance (a claim on the Chase page) | Met Every row reported as not checkable, and said so | 5/5 | 5/5 | none printed | 5/5 | 5/5 |
| misprint An amount that contradicts its own running balance | Met Flagged the contradicting row (row 31) and no other | 60/60 | 59/60 | 60/60 | 60/60 | 60/60 |
| scanned An image of a statement with no text layer | Met Refused as a scan; nothing sent anywhere | — | — | none printed | — | — |
What it has caught us getting wrong
The corpus exists to find these. Each one became a permanent test, and none is hidden once fixed.
- Numbers dropped from descriptions (found and fixed 13 September 2026). “TRANSFER FROM SAVINGS 4417” came out without the 4417, and a cheque number would have been lost the same way. Amounts and verdicts were unaffected, which is exactly why the arithmetic could not catch it: only comparing descriptions with ground truth did.
- A month at the start of every description (found and fixed 13 September 2026). On statements printing dates as “01 Apr”, where the date arrives as two pieces of text, every description began with “Apr”.
- A fee table that cost a whole statement its verification (found and fixed 13 September 2026, by drawing the Chase page's own claim as a statement). Fee rows printed after the closing balance were read as transactions, and because they carry no running balance, every row in the statement dropped to “not checked”. The claim on the page said they were excluded; until that day they were not.
- Posting dates glued to descriptions (found and fixed 13 September 2026, the same way). On a credit card statement with a transaction date and a posting date, the second date became the first word of every description.
- Every overdraft read as money in the bank (found and fixed 13 September 2026). A negative running balance printed as “-54.52”, “54.52 DR” or “(54.52)” was read as +54.52, flagging correct rows either side; printed “54.52 OD” it was not read at all, and the whole statement fell to “not checked”. The corpus had never contained an overdrawn account.
- Most of a busy day silently dropped (found and fixed 13 September 2026). On a statement that prints the date only on the first transaction of each day, every later transaction that day was skipped: four of seven rows gone.
- A month with one deposit lost the deposit (found and fixed 13 September 2026). A money-in column holding a single figure was not recognised as a column, so the only salary payment vanished — and on a statement printing its balance once a day, every sign was inverted as well.
- Newest-first statements flagged from top to bottom (found and fixed 14 September 2026). A statement listing its newest transaction first had every row flagged, because its running balance runs the other way. It is now put in ledger order when the arithmetic confirms that order, and the result says so.
- Entries with their figures on the last line lost (found and fixed 14 September 2026). Where a wrapped entry prints its amount on its final line, four of six transactions disappeared.
- A one-click fix that could turn a missing transaction into “Reconciled” (found and fixed 14 September 2026). When a row was missed, the next row was flagged and the first button set its amount to whatever made the balance add up: a card payment became a deposit, and the whole statement showed “Reconciled” with a transaction missing. A figure is now offered only when it is a plausible misreading of the printed one — a digit, a swap, a decimal point, money in and out — and the flag says which. Otherwise the row stays flagged and the page to check is named.
- A one-click “fix” that would have corrupted a correct amount (found and fixed 14 September 2026). A PDF holding two accounts was read as one ledger: the first row of the second account was flagged, and the suggested fix would have overwritten its correct amount. Each account is now checked against its own opening and closing balance.
- DR read as a heading (found and fixed 14 September 2026, by fuzzing: hundreds of random statements checked against ground truth). On a statement without column headings, the “CR” on an amount like “1,450.04 CR” was taken for a column heading, so the whole amount column became “money in” and every DR was read as a deposit.
- The wrong heading claiming a column (found and fixed 14 September 2026, the same way). Headings were matched in reading order, so “Money in”, 90pt away, could claim the money-out column before “Paid out” directly above it.
- Batches that crawled (found and fixed 13 September 2026, by the page-limit test rather than this corpus). Ten 500-page statements took six and a half minutes instead of eight seconds.
What it cannot tell you
- How a real bank's statement converts. Only real statements can. That is the next thing we are doing, and when there are results they go here, losses included.
- Everything about the reader in your browser. This corpus is scored on text positions extracted with pdfplumber. The browser uses pdf.js, which groups text differently. Separate end-to-end tests run the bank-shaped statements through pdf.js and compare every date and description, but this table is not those tests.
- Whether a description is right on your statement. Arithmetic proves amounts and balances. Nothing proves payee text except reading it, and the converter says so on every result.
Run it yourself
From a clean checkout of the source, with Node 22 and Python 3:
npm ci
pip install reportlab pdfplumber pikepdf pypdfium2 openpyxl
npm run fixtures # generate the corpus and its ground truth
npm run benchmark # score it; writes benchmark/results.jsonThe generator is scripts/make_fixtures.py and the scoring script is scripts/benchmark.ts. Continuous integration re-runs it on every change and fails if the results on this page are not the ones the current engine produces.