Same score. Twelve times the mistakes.

I tested 25 AI setups on 120 questions about a game's rulebook. Two scored almost the same, 99.1 and 98.9 out of 100. One invented a wrong rule in 2 answers, the other in 24. Pick by the score and you would never see that gap.

A study like this shows you which AI to trust with your work, how fast it answers, and where it goes wrong, before a customer finds out. A rulebook works like a policy manual: clear rules, plenty of tricky cases, and answers you can check. Swap in your refund policy or your onboarding guide and the test works the same way.

Book a Complimentary CallRead the report

What the study showed

What your AI reads matters more than which AI you pick

Without the rulebook passage in front of it, the best setup got 16 of 120 answers fully right. Given the exact passage, that same setup got 113 and answered almost three times faster. Even the weakest setup got 30 with the passage, nearly double the best score without it. All 25 scored higher and answered faster once they had it, so before you pay for a bigger model, check what your AI gets to read.

A clear bar cut 25 options to 3

A player waiting on a game master won't sit through 30 seconds. So I set the bar at a typical answer within 5 seconds, with more than half of all answers fully right. With the passage, 3 of the 25 setups cleared it, and all three were Gemma 4, a model family from Google.

The trade-off comes with numbers attached

The largest Gemma 4 model has one setting that lets it think before it answers. Off, a typical answer took 2 seconds and 96 of 120 were fully right. On, it took 6 seconds, just past the bar, and 113 were. That is 17 more right answers for 4 more seconds, a call you can make in one meeting.

What you get from a study on your AI

Your rules, real questions and examples replace the rulebook. Everything else stays the same.

  • A recommendation you can defend. Every option faces the same tests, and a plain-language report says which one I would pick and why.
  • Speed and usage for each option. You see how long every answer took and how many tokens it used, the unit most AI providers bill by.
  • Every mistake on the record. You can read what each AI said, what it got right and each hallucination, meaning every wrong or invented rule it added. Browse 900 graded answers from this study.
  • Tests you keep. Your team reruns them after every change, and a vendor update that breaks something shows up before your customers see it.

Bring one workflow and your rules. On the call we'll work out which decision to test first.

Book a Complimentary Call

How each answer is graded

Every question lists what a correct answer must say, worth 100 points in total, with partial credit for partial answers. Wrong and invented rules are recorded separately, which catches the hallucinations an average score hides. An answer counts as fully right only with every point earned and no such rule added. Here is a real question from the study with one graded answer, which scored 100 out of 100 and still failed.

case t03-001 · conditions, damage, and recoveryStudy 1

Your melee attack would reduce a conscious bandit to 0 Hit Points. You want to capture the bandit alive. What option do the general rules give you, and when does its effect end?

a correct answer must saypointsearned
Offers the optional knockout: the bandit drops to 1 HP and gains the Unconscious condition.4545
The bandit starts a Short Rest, and the condition ends when that rest ends.2525
Regaining any HP, or an action of first aid with a successful DC 10 Wisdom (Medicine) check, ends it early.3030
Wrong or invented rule · the answer says a Medicine check "restores HP". First aid ends the knockout without restoring HP.n/aadded
Granite 4.2 8B, thinking off, passage supplied · score 100 / 100 · fully right no1 hallucination

How Study 1 was run

The suite started with my own product, a storytelling engine: an agent that can create campaigns and run games. Each design decision that matters gets a benchmark before it gets built. It is the method I bring to a client's product: find the decisions that matter, and measure them.

Agents drafted the 120 questions from SRD 5.2.1, the open rulebook of Dungeons & Dragons, under my direction. I reviewed 18 of them in full and checked the rest. Every model ran locally on a single desktop graphics card, with and without the passage. A grader model scored each answer against the question's credit rules; I reviewed its flagged judgements and fed the corrections back into the grading guides.

questions120 across 12 topics, from SRD 5.2.1
AI setups15 open models in 25 setups, 2.7B to 31B parameters
conditionswith and without the rulebook passage
answers6,000 graded
gradingpoints for required claims, wrong and invented rules tracked apart
hardwareone RTX 3090, all answers generated locally

The totals on this page cover all 120 questions. The 18 public questions and their 900 graded answers are open to inspect. Study 2 is under way on the same 120 questions, measuring how much of the gain from the passage survives when a keyword search has to find it.

Read the reportExplore the answers

This work includes material from the System Reference Document 5.2.1 ("SRD 5.2.1") by Wizards of the Coast LLC, available at https://www.dndbeyond.com/srd. The SRD 5.2.1 is licensed under the Creative Commons Attribution 4.0 International License, available at https://creativecommons.org/licenses/by/4.0/legalcode.