Grading a tax search engine against the tax authority's answers

2026 · Backend, search and evals

I built a search engine for Mexico's tax law that answers in plain Spanish, and graded it against the answers the tax authority itself publishes.

Cloudflare Workers, D1, Vectorize, Workers AI, GraphQL Yoga, DataLoader, TypeScript, Claude

TL;DR On October 15 I started fundamento, which reads ten documents of Mexico’s federal tax law, 2,577 articles and rules, and the 6,342 citations between them. You ask in plain Spanish, it finds the articles, and Claude answers from them with every citation checked. I graded every search and answer setup on 189 questions built from criteria the tax authority publishes. The live search puts a cited law article in the top five for 80.4% of them, and a Claude reranker that tied it didn’t ship. The code is public.


Every line in the circle below is one citation between two articles of Mexican tax law. Four federal tax laws, the 2026 income law, their regulations and the Miscelánea, the tax authority’s 1,215 rules for the year, cite each other 6,342 times.

A circle of dots, one per article or rule, in arcs labelled LISR, RLISR, LIVA, RLIVA, LIEPS, RLIEPS, CFF, RCFF, LIF and RMF, under three toggles named Leyes, Reglamentos and Miscelánea. Maroon lines from the RMF arc on the left fill most of the circle and gather at LISR at the top and CFF at the bottom right, and blue lines join each regulation to the law beside it.
The Miscelánea's rules, in maroon, make 4,174 of the citations, 1,338 to the tax code (CFF) and 1,013 to the income tax law (LISR).

I wanted a backend project with a real graph in it and a test set I didn’t write myself. Tax law has both. The laws cite each other constantly, and SAT, the tax authority, publishes criteria on how it reads them. Each criterion cites the articles it interprets, so each one is a question with a known answer.

An answer key from the tax authority

The criteria gave me 189 questions, 133 from Anexo 7 to choose methods with and 56 from Anexo 3 that I held out. Nobody types “Deducible del seguro de gastos médicos. No es una deducción personal.” into a search box, so a language model rewrote each criterion the way a taxpayer would ask it, leaving the answer out. That one became “¿Puedo deducir del ISR el deducible que pago de mi seguro de gastos médicos?” An answer left in would leak, the way it did in QAnelita’s first eval.

Every comparison is paired. Two methods answer the same resampled questions, and the gap between them gets a 95% interval, so a two-point win that could be noise reads as noise. The first fight was keywords against meaning. Restricted to the laws, keyword search found a cited article in the top five for 47.1% of the plain questions, and vectors from embeddinggemma for 75.7%, a gain of 21.7 to 36.0 points.

Rule numbers in the margin

The Miscelánea’s PDF prints each rule’s number in a margin column. Read as plain text through PDFKit, a page’s numbers bunch up at its top, glued to the first line, which here belongs to rule 2.1.13.

2.1.14. 2.1.15. obtenidos con motivo de la celebración del mismo convenio durante el mismo periodo; el

So the parser reads each line with its page, x and y in points, and text.

46	128	509	CFF 16, 17-B, LISR 17, RMF 3.21.2.1.
46	128	493	Documentación en copia simple
46	71	476	2.1.14. Para los efectos de los artículos 18, 18-A y 19 del CFF, los documentos que se acompañen a

Lines inside a paragraph sit 14 points apart or less and paragraphs 16 or more, and none of the 37,734 gaps rounds to 15. So paragraphs come from the gaps, and a rule’s title is the paragraph above its number. PDFKit reports no font weight for the bold titles, so the layout found them, for all but 8 of the 1,215 rules. Each rule’s closing line of references, like the one above that title, becomes links.

Fundamento's page for rule 2.1.14: the rule number in maroon above the title Documentación en copia simple, its title and chapter in the Miscelánea, a plain-Spanish summary headed En pocas palabras, and a side list headed Esta regla cita with CFF 18, 18-A and 19.
Rule 2.1.14, titled from the paragraph above its number, with the three articles it cites beside a plain-Spanish summary.

Laws first, rules second

When the corpus grew from three laws to ten documents, the Miscelánea’s rules, which paraphrase the law, crowded it out of one shared ranking. On the same 173 questions as before, first-place hits fell from 45.7% to 29.5%, and rules took 1.6 of each top five on average. So search returns two lists, the laws first and then the regulations and rules, which is also their legal order. The law list scores 45.5% in first place on all 189 questions, 9.5 to 19.0 points above one ranking.

Asked “¿Puedo deducir mis lentes graduados?”, the law list leads with CFF 29-A and LISR 27, and the second list opens with article 264 of the ISR regulation, which caps prescription glasses at 2,500 pesos a person a year.

Fit the free plan

Cloudflare’s free plan stores 5 million vector dimensions. The ten documents make 7,111 vectors, about 5.5 million dimensions at embeddinggemma’s 768 and 3.6 million at 512. The model is trained so its first dimensions stand on their own, and cut to 512 it scored 0.5 to 7.9 points higher at top five.

The plan also rations Workers AI by the day, and partway through the build it stopped answering. Since embeddinggemma’s weights are open, I embedded the 5,045 new texts on my laptop in 5.5 minutes, and they matched Cloudflare’s vectors at a cosine of 0.9993 or better. Production still embeds each search query on Workers AI. Vectorize’s approximate search ranked all 189 questions the way exact search did, the same kind of check my screenshot replays pass against live runs.

57 queries down to 6

The API is GraphQL because the data is a graph. Fetching citations and criteria for ten search hits took a median of 57 D1 queries. DataLoader batches them into 6, and measured live on the first three laws, the median search went from 587 ms to 329 ms. Every response reports its query count, so anyone can check it in /graphql.

Claude rewrites the question

Fusing keyword search into the vectors made results worse, dropping the first result by 1.6 to 13.2 points. A taxpayer and the law rarely use the same words. So before searching, Claude Haiku rewrites each question in the law’s vocabulary, as terms like “plazo declaración anual personas físicas abril”, and keyword search runs on that.

Fused with the vectors, the rewrite beat the vectors by 1.6 to 7.9 points at top five, 80.4% against 75.7%, and that’s what runs live. It also covers the bad days. When Workers AI is out, keyword search on the rewrite still reaches 67.7% at top five, where plain keywords got 47.1%.

A reranker that tied

Vectors are the one part of a search that needs Workers AI, so I tried letting Claude do their job. Keyword search on the rewrite pulled 30 articles, and Claude Haiku read them and picked the best ten. On the tax questions it put a cited law in the top five 83.1% of the time against 80.4% live, a paired gap of −4.2 to +8.5 points, so a tie. On the traffic site’s 28 questions it fell from 100% to 85.7%. A tie that adds a second model call to every search doesn’t ship, so the vectors stayed.

Answers beside the law

Answers cite their sources in brackets, and any citation the model wasn’t given is removed before the answer leaves the Worker. Claude Haiku answered each question and Claude Opus graded the answer against the question’s criterion. Reading only laws, regulations and rules, 22.2% of answers matched SAT and 6.3% contradicted it. Most of the rest said the passages don’t settle the question, which statutes rarely do. With every criterion searchable, as on the live site, Claude Haiku matched SAT 87.3% of the time and Claude Sonnet 91.0%, with 2 wrong answers instead of 5, so Sonnet answers live. Both are upper bounds, since each question was written from its own criterion.

The answer started as a box next to the results. Now it’s a panel that stays open while you move around the site. Whatever you search, the law lists on the left and SATGPT, the site’s assistant, answers on the right, and every article it cites is marked in the list. You can keep asking, and a follow-up like “¿y si soy persona moral?” gets rewritten into one self-contained search before Claude answers it, since on its own it matches nothing useful.

Fundamento at desktop width. On the left, the question ¿Puedo deducir mis gastos médicos en la declaración anual? as the page title, a similar frequent question, and the laws found, with article 151 of the income tax law marked Citado en la respuesta. On the right, the SATGPT panel answers in maroon bold that medical expenses are a personal deduction in the annual return, cites Art. 151 LISR and Art. 264 RLISR, and lists the conditions as bullets.
A search on the live site: the laws on the left, SATGPT's answer on the right, and the article it cites marked in the list.

Every article in force also gets a few plain-Spanish sentences on top, written by Claude Sonnet from the official text, and three questions that open a search with SATGPT’s answer. The official text stays right below, and it’s the one that counts. Claude Opus checked 40 summaries on each site against the text. One of the 80 left out a condition, and 5 of their 240 questions went beyond what the article says. That’s why the questions go to SATGPT, which searches the whole law, instead of standing as answers.

The En pocas palabras block on article 27 of the income tax law: a paragraph on the requirements for companies' authorized deductions, including the 7% and 4% limits on donations and bank payment above $2,000.00, three questions under Pregúntale a SATGPT, and a note that Claude Sonnet wrote it from the official text, which starts right below. A side list shows the 15 articles it cites and the 84 that cite it.
Article 27 of the income tax law, with its summary on top of the official text.

If I started over

I’d collect real questions as early as the criteria. Questions written from SAT’s own answers make every number here easier to hit than a question SAT never addressed, which is why the 91.0% is an upper bound.

I’d also read every PDF by position from the first day, instead of finding out from the Miscelánea’s margin.

Everything that differs between the two sites lives in one config, so the same engine now runs Maneja Sin Broncas, Mexico City’s traffic rules with every fine in 2026 pesos, at manejasinbroncas.carlosaguirre.workers.dev. Try fundamento at fundamento.carlosaguirre.workers.dev, or read the code on GitHub.