QandyEval: real financial work, graded.
QandyEval is QANDY's benchmark for financial agents. It measures whether a model can do the financial work people actually need done, under the rules in force on the day, in five domains and nine jurisdictions.
The financial work people actually need done.
-
Personal tax
Payroll withholding, annual returns, eligibility, residency and cross-border income, investment income, pensions, and the return forms themselves.
-
Corporate tax & accounting
Taxable income, rates and regimes, indirect tax returns, invoice compliance, withholding and treaties, and company bookkeeping through to the period-end close.
-
Save
Every contract and invoice in a household: what each costs, when it can end, the best offer for real usage, and cancelling or switching on the user’s behalf.
-
Trading
Order and market rules, costs and after-tax profit and loss, product access and suitability, disclosures, and margin and risk maths.
-
Insurance
Application and claim forms, policy extraction, coverage decisions, settlement calculations and public health reimbursement, across 11 lines of insurance.
- United States
- European Union
- United Kingdom
- Mainland China
- Japan
- South Korea
- Hong Kong SAR
- Taiwan
- Canada
The EU is covered at EU level and per member state, in tiers, across all 27 member states.
Right numbers, right forms, right law, every run.
-
Numbers and real forms
Every domain is tested on two things: getting the numbers right, and filling in the official forms field by field, consistent across forms.
-
The law on the day
Every item states the date it applies to. The hardest tier is built on 2026 rule changes and refreshed every quarter, so stale knowledge shows.
-
Audited answer keys
Answer keys come from calculation engines audited against primary sources.
-
Right every time
An item only counts if the model gets it right in every run (pass^k). Scores are 0–100 with a 95% range.
-
Simulated, never real
Portals, inboxes and drives are simulated. Nothing is ever submitted to a real tax authority or insurer, and no personal data enters the eval set.
Acting for the user, only with their authority.
Most agent evals fail an agent for taking any outward action. In Save, cancelling and switching is the job. So the agent gets written authority: cancel these, switch these if the saving is over a set amount, ask about the rest.
You ask. You approve. QANDY SAVE executes.
- 01The final state of every simulated provider portal.
- 02Every action stayed within the authority the user gave, and was sent in time.
- 03The effective date is right, with no gap in cover or service.
- 04Nothing else was touched.
- 05Decisions outside that authority went back to the user as a question.
QANDY 2.0 27B against Claude Opus 5.5 and GPT 6.1 Sol.
All three models will answer the same frozen item set under the same rules, scored 0–100 in every domain.
- QANDY 2.0 27B
- Claude Opus 5.5
- GPT 6.1 Sol
Results pending. Scores will be published here once all three models have been run on the same frozen item set.
Show as table
| Domain | QANDY 2.0 27B | Claude Opus 5.5 | GPT 6.1 Sol |
|---|---|---|---|
| Personal tax | Pending | Pending | Pending |
| Corporate tax & accounting | Pending | Pending | Pending |
| Save | Pending | Pending | Pending |
| Trading | Pending | Pending | Pending |
| Insurance | Pending | Pending | Pending |
QANDY 2.0 27B runs on your own computer.
It powers QANDY SAVE, a personal financial agent that finds where you overpay and handles the switch, but only after you approve it.