Mortgage-Bench is a synthetic benchmark of 150 cases that tests AI agents on the whole processor workflow: gather and cite evidence, reconcile conflicts, flag risks, assess readiness and recommend Approve, Suspend or Deny. No model tested passed half the cases.

Read the PaperExplore Code· Private repository, open to reviewers on request

Published: October 2026 · Version 2
Authors: Nathan Sekar, João Aires de Matos, Michael Vandi
Models evaluated: 4 models in August, 3 in October, plus 3 reference agents
Data: Fully synthetic; no personal data

Scope

A reproducible test of the whole processor workflow

150

Synthetic cases

Seeded by a generator: ten for each program and difficulty level. Gold decisions: 50 Approve, 60 Suspend, 40 Deny.

Loan programs

FHA, VA, USDA, Jumbo and DSCR, each with synthetic rules that mimic the workflow, not official policy.

Difficulty levels

Beginner, Intermediate and Advanced, rising with the complexity and unreliability of the evidence.

Why it exists

Lending needs agents that can justify a decision, not just extract a field

A mortgage processor combines income, asset, credit, employment and property documents into a recommendation that must rest on a documented ability-to-repay assessment and on program rules from FHA, VA, USDA and private investors.

Existing financial benchmarks test isolated extraction or question answering. The closest lending benchmark, MortarBench, tests single-turn questions over synthetic bank statements. No public benchmark covered the full processor workflow.

Gather and cite

Complete the application and cite the source of every entered value.

Reconcile

Resolve conflicts between documents, email and the verification portal.

Assess

Decide readiness, record conditions and select the applicable risk flags.

Decide

Recommend Approve, Suspend or Deny with a rationale that is scored with the label.

Benchmark overview

Every program, at every level of difficulty

Programs differ in who sets the rules and what evidence decides eligibility. Difficulty rises with conflicts, edge cases and noise.

5 programs × 3 difficulty levels × 10 cases

Select a cell to see what the program tests and what makes the level hard. Gold decisions across all 150 cases: 50 Approve, 60 Suspend, 40 Deny.

Beginner

Intermediate

Advanced

FHA

USDA

Jumbo

DSCR

VAIntermediate

Program · VA Pamphlet 26-7

Residual income test alongside DTI; a ratio above 41% calls for closer scrutiny unless residual income exceeds the guideline by at least 20%; no VA minimum credit score.

Difficulty

Conflicts between sources, such as an email that disagrees with a pay stub.

October reruns · pass rate

Pass rate (mean score) for all VA cases (90 attempts) and all Intermediate cases (150 attempts). Results are reported per program and per level, not per cell.
Model	VA	Intermediate
Claude Sonnet 5.5	72% (0.81)	68% (0.79)
GPT-5.6 Luna	63% (0.77)	84% (0.84)
Gemini 3.8 Flash	0% (0.70)	0% (0.63)

Synthetic rules and thresholds; no case states any program's actual policy.

Workflow

From documents to a documented recommendation

Each case runs seven steps across five information surfaces. Answer keys are hidden from the agent.

1
Documents and evidence
- 1Identify the loan program
- 2Gather evidence from the five surfaces
2
Extraction and reconciliation
- 3Complete required fields and cite each source
- 4Reconcile conflicts to the verified source
3
Readiness and risk
- 5Assess readiness and record conditions
- 6Select the applicable risk flags
4
Recommendation and rationale
- 7Approve, Suspend or Deny with a rationale, then submit

Loan package

Identity, income and tax documents, assets, property, credit and program forms

Primary evidence

Email inbox

Messages from the borrower and third parties

May confirm or contradict a document

Verification portal

Verified values for designated fields

Authoritative when sources conflict

Application form

Required application fields

The agent enters each value and its source

Underwriting screen

Readiness, conditions, risk flags, decision and rationale

Records the assessment and recommendation

Results

Agents handle evidence better than judgment

Pass rate and decision accuracy are the primary metrics. A case passes only when its full workflow scores at least 0.80; decision accuracy counts the Approve, Suspend or Deny label alone.

The two runs are reported separately and not pooled.

Current models: pass rate vs. decision accuracy

Shares of 450 attempts per model on the same 150 cases.

Gemini 3.8 Flash had the highest decision accuracy but passed no case: it never chose Deny, and its field-value score (0.646) was far below the others'. GPT-5.6 Luna chose Suspend in 422 of 450 attempts.

Source: Sekar, Aires de Matos and Vandi, Mortgage-Bench, Version 2 (Addy AI, October 2026). Table 8.

View data table: Current models: pass rate vs. decision accuracy

End-to-end pass rate and decision accuracy, in percent.
Agent	End-to-end pass rate	Decision accuracy
Claude Sonnet 5.5	37.8%	43.8%
GPT-5.6 Luna	38%	42.2%
Gemini 3.8 Flash	0%	52%

Full results, October reruns

Mean score is the weighted case score from 0 to 1. With 150 cases, a pass-rate interval near 40% spans about ±8 points.

October 2026 reruns, three tries per case (450 attempts per model). Intervals resample the 150 cases with each case's three tries kept together. pass^3 is the share of cases passed on all three tries. Costs are token-based estimates at list prices.
Agent	Mean score (95% interval)	Pass rate, S ≥ 0.80	Decision accuracy	pass^3	API cost, 450 attempts
Claude Sonnet 5.5claude-sonnet-5-5 · temperature not supported	0.662 (0.635–0.689)	37.8% (31–44%)	43.8% (197/450)	23.3% (35/150)	$14.44
GPT-5.6 Lunagpt-5.6-luna · temperature not supported	0.656 (0.628–0.684)	38.0% (31–45%)	42.2% (190/450)	27.3% (41/150)	$0.71
Gemini 3.8 Flashgemini-3.8-flash · temperature 0	0.635 (0.615–0.655)	0.0%	52.0% (234/450)	0.0%	$2.00

Passing was not stable across tries: Sonnet 5.5 passed 35 cases on all three tries and 44 on only one or two; Luna passed 41 and 32.

Skill-level scores, October reruns

Mean component score from 0 to 1, 450 attempts per model. Extraction, citation and reconciliation were strong; readiness and the final decision were weak.

Source: Sekar, Aires de Matos and Vandi, Mortgage-Bench, Version 2 (Addy AI, October 2026). Table A1.

View data table: Skill-level scores, October reruns

Mean component scores for the October reruns, by model.
Component	Claude Sonnet 5.5	GPT-5.6 Luna	Gemini 3.8 Flash
Application-field accuracy (25%)	0.923	0.932	0.646
Source-citation correctness (20%)	0.999	0.993	1.000
Risk-flag selection (15%)	0.825	0.797	0.676
Income and conflict reconciliation (10%)	1.000	0.998	0.904
Readiness and conditions (10%)	0.425	0.471	0.297
Final decision with rationale (20%)	0.524	0.522	0.594

Where decisions went wrong

Rows are the gold decision, columns the agent's decision. Over-suspension was the most common error: the current models rarely deny and send most gold-Approve files to Suspend.

Document heuristic

1 run, 150 cases

Gold ↓

Approve

Suspend

Deny

Approve

Suspend

Deny

Claude Sonnet 5.5

3 tries, 450 attempts

Gold ↓

Approve

Suspend

Deny

Approve

110

Suspend

152

Deny

115

GPT-5.6 Luna

3 tries, 450 attempts

Gold ↓

Approve

Suspend

Deny

Approve

132

Suspend

171

Deny

119

Gemini 3.8 Flash

3 tries, 450 attempts

Gold ↓

Approve

Suspend

Deny

Approve

117

Suspend

117

Deny

120

Source: Sekar, Aires de Matos and Vandi, Mortgage-Bench, Version 2 (Addy AI, October 2026). Table A2. The original model runs did not store wrong labels.

Pass rate by difficulty level

150 attempts per level. Intermediate cases had the highest pass rates, matching the August pattern for Claude Sonnet 4.5.

Values show pass rate · mean score.

Source: Sekar, Aires de Matos and Vandi, Mortgage-Bench, Version 2 (Addy AI, October 2026). Table A3.

View data table: Pass rate by difficulty level

Pass rate and mean score for the October reruns.
Difficulty	Claude Sonnet 5.5	GPT-5.6 Luna	Gemini 3.8 Flash
Beginner	23% · 0.60	12% · 0.55	0% · 0.72
Intermediate	68% · 0.79	84% · 0.84	0% · 0.63
Advanced	23% · 0.60	18% · 0.58	0% · 0.56

Methodology

Scoring and evaluation environment

A rule-based grader scores every case the same way for every agent, including the reference agents.

Weighted score

Six weighted components

S = Σ w_k s_k, each component between 0 and 1. The decision is graded by rules, with no LLM judge.

Scoring components, weights and matching rules.
Component	Weight	Matching rule
Application-field accuracy	25%	Money within one cent; other fields match ignoring case
Source-citation correctness	20%	The cited source is correct and the agent opened it
Final decision with rationale	20%	Rule-based, no LLM judge: 70% label, 30% overlap with gold reason tags
Risk-flag selection	15%	Overlap with the gold flag set; an empty set matches an empty set
Income and conflict reconciliation	10%	If incomes conflict, the agent must open and enter the verified income
Readiness and conditions	10%	Ready only when no conditions remain

Caps and the pass line

0.80

Pass threshold

A case passes when its weighted score S is at least 0.80.

0.50

Wrong decision cap

A wrong Approve, Suspend or Deny caps the score, so evidence handling alone cannot pass.

0.79

Wrong critical field

A wrong critical field caps the score just below the pass line.

0.69

Uncited critical field

An uncited critical field caps the score lower still.

With a correct decision, an agent still needs 75% of the remaining credit to pass. Weights and caps were set by design.

Evaluation environment

All agents received the same instructions, tools and 150 cases, with no skipping or selective reruns. Original runs (August 11–12, 2026): Claude Sonnet 4.5, GPT-4o and Gemini 2.5 Flash at temperature 0, and GPT-5.6 Luna, which rejects the temperature parameter, one try per case. October 7–8, 2026 reruns: Claude Sonnet 5.5, GPT-5.6 Luna and Gemini 3.8 Flash, three tries per case; only Gemini accepts a temperature, set to 0. Gemini 2.5 Flash ran through Vertex AI.

Temperature 0 is not deterministic on hosted models. Costs are list prices on the run dates. GPT-6 and open-weight models were not tested.

Reference agents

Three agents without a language model bound the score range. The oracle copies the answer key (1.00, every case passed). The naive agent enters the gold field values, cites nothing, marks every file ready and approves every loan (0.36, no passes). The document heuristic reads the visible documents with simple patterns and applies the benchmark's own thresholds for credit score, LTV, reserves and DSCR, but does not compute DTI; it never sees the answer key (0.667, no passes).

Statistics

Mean scores have 95% intervals from a case bootstrap (1,000 resamples, 2.5th and 97.5th percentiles, seed 4219); for the reruns, each case's three tries are resampled together. Pass rates use Wilson intervals for single runs and the same case bootstrap for the reruns. Paired comparisons are left to future work.

Quality assurance

Each case was checked for solvability, internal consistency and answer leakage; the oracle scored 1.00 and the naive agent 0.36. The cutoffs are synthetic, and the cases and rubric have not had a recorded expert review.

Reproducibility

Paper, code and data

Reviewers rerun the benchmark with their own API keys. Links appear here as each artifact is released.

Paper (PDF)

Version 2 · October 2026 · 17 pages

Open

Code and harness

Generator, grader, harness, prompts and all run outputs

Private repository, open to reviewers on request

Cases and answer keys

150 cases, suite mortgage-bench-development-v1, version 1.0.0

Shared with reviewers on request

DOI

Persistent identifier

No DOI assigned yet

The generator, the 150 cases, answer keys, grader, harness, prompts and all run outputs are in a private Addy AI repository open to reviewers. To request access, contact the research team.

Read before citing

Limitations and disclosure

Mortgage-Bench is a research benchmark developed by Addy AI. Read these limitations before citing any result.

SyntheticEvery case, document and threshold is synthetic. No borrower data is used.

Developed by Addy AIBuilt, funded, owned and licensed by Addy AI. Not independently validated.

Not a compliance testIt does not assess regulatory or legal compliance.

Synthetic data

Synthetic rules and documents miss real-world complexity and may be easier than real files. The cutoffs are synthetic, and the cases do not state any program's actual policy.

Not a compliance test

Mortgage-Bench is a research benchmark. It does not assess regulatory or legal compliance, and results are no evidence of compliance.

Developed by Addy AI

Addy AI built the benchmark, owns and licenses it, funded the work and sells mortgage AI software. None of the evaluated agents is an Addy product.

Not independently validated

The cases and rubric have not had a recorded expert review, and agreement between the rule-based grader and expert judgment is not yet reported.

Decision rules not shown to agents

Agents were not shown the credit-score, LTV and DTI cutoffs that set most gold decisions, and the prompt told them to prefer Suspend when issues remained. Decision accuracy partly measures whether an agent can infer unstated cutoffs.

Evaluation design

One prompt; one run per case in August and three in October. Hosted models are not deterministic. GPT-6 and open-weight models were not tested.

Scoring choices

Weights and caps were set by design; sensitivity analyses, error taxonomy, calibration and paired significance tests are not yet reported.

Scope, fairness and contamination

U.S. mortgage workflows only. Disparate treatment was not tested. Public cases may enter training data; a private held-out split would prevent this.

Cite

How to cite Mortgage-Bench

Mortgage-Bench: A Benchmark for End-to-End Mortgage Loan Processing Agents

Nathan Sekar, João Aires de Matos, Michael Vandi · October 2026 · Version 2 · 17 pages

Download PDF

Citation

Sekar N, Aires de Matos J, Vandi M. Mortgage-Bench: A Benchmark for End-to-End Mortgage Loan Processing Agents. Version 2. Addy AI; October 2026.

BibTeX

@techreport{sekar2026mortgagebench,
  title       = {Mortgage-Bench: A Benchmark for End-to-End Mortgage Loan Processing Agents},
  author      = {Sekar, Nathan and Aires de Matos, Jo{\~a}o and Vandi, Michael},
  institution = {Addy AI},
  year        = {2026},
  month       = oct,
  note        = {Version 2}
}