FinVerify Research — Deterministic Verification Layer
Financial AI is confident.
It is not always right.
A verification layer that binds every numeric claim to its source, before it becomes a decision.
Prompt
What was AutoZone's operating margin in FY2019?
Model answer
AutoZone reported an operating margin of 23.4% in fiscal 2019.
Verification pipeline
deterministic- Extract claim01
- Retrieve evidence02
- Detect mismatch03
- Repair04
- Verify05
Awaiting verification
The verified answer will bind here once the pipeline completes.
Verification requests
0
processed
Claims verified
0.0%
evidence-backed
Average latency
0 ms
per claim
Evidence sources
10-Ks · 10-Qs · Calls
SEC filings
Open source
GitHub
MIT license
§ 01 · Problem
A wrong number,
stated with full
certainty.
A financial LLM will compute a company's free cash flow, miscalculate the arithmetic by a factor of ten, and present the result in the same authoritative register as its correct answers. There is no uncertainty token. No visible seam. No downstream check.
In consumer chat, that's an annoyance. In credit, in equity research, in treasury, in risk — it's a decision made on a fabricated number.
User
From the attached 10-K, what was the company's FY2022 free cash flow?
Model
"The company generated $18.4 billion in free cash flow in fiscal 2022, driven by strong operating performance and disciplined capital expenditure."
→ Source states $1.84B. Off by 10×. Never surfaced to the user.
- Silent
- numerical errors
- 0
- downstream flags
- 100%
- model confidence
§ 02 · Diagnosis
Why existing financial AI
fails quietly.
Large models are optimised to produce fluent text, not to preserve arithmetic identity. Three failure modes recur in every financial deployment we have studied.
Millions and billions are interchanged mid-answer. Basis points drift into percentage points. Scale errors pass unnoticed.
Numeric claims are generated from parametric memory, not read from the cited document. The citation is decorative.
When the answer is wrong, nothing in the response distinguishes it from a correct one. There is no seam for a downstream system to catch.
§ 03 · Method
The Deterministic Verification Layer. A numerical checker for models that reason over money.
Extract
Every numeric claim in the model's answer is parsed with its units, scale, and context — as structured propositions, not free text.
Bind to evidence
Each claim is bound to a passage in the source document. The DVL derives an evidence band — internally, UNIT_BANDS — from the source itself. It does not see the ground-truth answer.
Verify, deterministically
Claims are checked with arithmetic and unit logic — not by asking another LLM whether the first one was probably right. Fail states are inspectable, not vibes.
Results · FinQA · Mistral-7B-Instruct (fine-tuned)
Adding the DVL — without touching model weights, without more data, without a larger model — lifted execution accuracy from near-zero to state-of-the-art range for the parameter class.
execution accuracy · FinQA test split · Δ = +41.6pp
§ 07 · Verification in numbers
What verification
actually delivers.
Measured on the FinVerifyBench evaluation harness, Mistral-7B-Instruct (fine-tuned) with the Deterministic Verification Layer, against the same base model without it.
Figure 04a
Execution accuracy · FinQA
42×
Improvement over the unverified baseline (~1% → 42.61%). Same weights, same data, same model class.
Figure 04b
Verification latency · p50
214 ms
Median end-to-end DVL trace time, measured on FinVerifyBench with cold-start caches disabled.
Figure 04c
Numeric claims · evidence-bound
100%
On the FinVerifyBench test split. Every numeric claim in a verified answer is bound to a passage in the source document — or it does not ship.
1,070
benchmark questions
3
source datasets · FinQA · TAT-QA · FV-Synth-1
Open
reference implementation · CC-BY 4.0
Deterministic
verification pipeline · reproducible traces
Methodology · FinVerifyBench v0.4 · commit 7f21-8ac · results reproduced 2025-11-14
§ 04 · Product
Every claim, every source,
every time.
The DVL runs as an inference-time layer between a financial LLM and its output. Questions in, evidence-bound answers out — every number cited, verified, and inspectable.
Open the live demo →$ dvl query --doc AAPL_10K_2023.pdf
› What was Apple's gross margin in FY2023?
[extract] claim: gross_margin(FY2023) = ?
[retrieve] Item 7 · MD&A · p. 27, line 14–18
[bind] UNIT_BANDS = [44.10%, 44.15%]
[model] proposed: 44.13%
[verify] in-band · ACCEPT
$ _
Verified response
- Inline source citation on every numeric claim
- Auto-repair when a claim falls outside the evidence band
- Full audit trace — reproducible, not probabilistic
Trust isn't generated.
It's verified.
§ 05 · Benchmark
FinVerifyBench — a benchmark for the right answer, for the right reasons.
Most financial QA benchmarks reward the final number and forget everything else. FinVerifyBench separates models that reason correctly from models that get lucky — by scoring evidence binding, unit consistency, and repair behavior alongside answer accuracy.
| Source | Samples | Domain | What it catches |
|---|---|---|---|
| FinQA | 512 | Earnings reports · SEC filings | Multi-step numerical reasoning over tables + text |
| TAT-QA | 341 | Hybrid tabular / textual financial QA | Cross-cell arithmetic, unit mismatches |
| Synthetic — FV-Synth-1 | 217 | Constructed adversarial reasoning tasks | Unit swaps, scale shifts (M ↔ B), sign flips |
| Total | 1,070 | Public · versioned · CC-BY 4.0 | |
- Baseline0%
- + Unit checks0%
- + Evidence binding0%
- + Arithmetic repair0%
- + Full DVL0.00%
Same weights · same data · verification composed additively
§ 06 · Infrastructure
Built for organizations that
cannot afford incorrect numbers.
Verification runs at inference time — a deterministic layer between the model and the decision. Bloomberg Terminal discipline. Stripe-grade infrastructure.
Research
Peer-reviewed methodology, published preprints, and reproducible traces.
Benchmarks
FinVerifyBench — 1,070 questions across FinQA, TAT-QA, and FV-Synth-1.
Documentation
Architecture notes, integration guides, and inspection primitives for the DVL.
API
Inference-time verification endpoint. Questions in, evidence-bound answers out.
§ 06 · Publications
A verification layer nobody can audit is not a verification layer.
The FinVerifyBench dataset, evaluation harness, and DVL reference implementation are published in the open. The methodology is inspectable end-to-end — because trust in financial infrastructure is earned through disclosure, not asserted through a badge.
Aaditya Thokal
Founder • Research Lead
FinVerify Research Lab
§ 08 · Founder
Meet the Founder
Aaditya Thokal
Founder • Research Lead
Aaditya leads FinVerify's research into trustworthy financial AI. His work focuses on deterministic verification, evidence-grounded numerical reasoning, Financial Constraint Graphs (FCGs), and reproducible evaluation through FinVerifyBench. The goal is to build open infrastructure that enables researchers, developers, and enterprises to build reliable, transparent, and verifiable financial AI systems.
Get in touch
§ 09 · Live demo
Watch a hallucinated number
get caught in real time.
The live demo runs an unmodified fine-tuned Mistral-7B behind the DVL. Ask a question about a real 10-K. Watch the trace. Read the verified answer.