RAG vs JAG: when you know the questions, you don't need embeddings

Retrieval-augmented generation (RAG) works the same way almost everywhere. You embed every page of a document, you embed the question, you take the pages whose vectors sit closest to the question's, and you hand those to an LLM to answer from. [Ragextract](https://ragextract.com) does exactly this today: each page is embedded as an image, and each question pulls back the six pages that look most like it. That design assumes you don't know the questions in advance. Search has to be ready for anything, so you index everything once and compare vectors at query time. Ragextract is different in one respect. Its users build a table, and the table's columns are the questions. *What is the total on this invoice? Who signed the agreement? When can it be terminated?* Those columns exist before the documents are uploaded. So we asked a simple question: **if you already know the questions, why search at all?** Why not ask a model to read each page and judge, for each question, whether the answer is on it? We've been calling this **JAG: Judgement-Augmented Generation**. ::: info Read this first This is specific to Ragextract, where the questions are table columns written before any document arrives. It's an experiment in improving how well Ragextract answers them, not a claim that RAG is obsolete. Nothing here has shipped. If your questions arrive at query time, such as in a chatbot or an open search box, you still need RAG. ::: ## RAG and JAG **RAG** compares vectors. The page and the question are turned into embeddings separately, and similarity decides which pages the LLM sees. It always returns its top results, whether or not any of them is relevant. **JAG** asks for a judgement. A *decision model* looks at a page image and a question together and returns a probability that the page answers it. This kind of model returns a calibrated yes/no instead of generated text, which makes it fast and cheap. One request per page can carry every column at once, so the page image is paid for once however many questions ride along. Rank the pages by that probability, and the LLM reads the best ones. And because the score means something on its own, a document where no page scores well can be marked *not found* without asking the LLM at all. We tested two shapes: - **JAG:** every page judged against every question. No embeddings, no vector store. - **JAG rerank:** RAG shortlists 15 pages, and the judge reorders them and decides whether any answers. Embeddings stay, so search stays. ## The test We reused the bundle from our [document-splitting post](/blog/automation/n8n-decisions-api-document-splitting): a synthetic legal discovery dump from a made-up dispute, *Ecorp Group v Acme Electric Company*. It's a single 300-page PDF holding 167 documents: emails and threads, letters, invoices, meeting minutes, meter-reading printouts, site visit reports, price-change notices, a supply agreement and its amendment, and handwritten call notes. Many pages are scanned, some are sideways, and none has a text layer. Every person, company and figure in it is invented. We treated the whole bundle as one upload and wrote 30 column-style questions for it: - **20 answerable**, across four kinds: - **Exact:** an identifier or date, such as *"What is the total amount due on invoice AEC-2025-104276?"* - **Table:** a value in a table or schedule, such as *"What new day unit rate takes effect from 1 September 2024?"* - **Visual:** handwriting, signatures or photos, such as *"What did Tom note down on his call about Ecorp Warehouse 3?"* - **Semantic:** a finding or reason, such as *"What reason does Acme give for the price change taking effect on 1 December 2024?"* - **10 absent**, with no answer anywhere in the bundle. Some are obvious, like gas rates in an electricity dispute. Others are traps: an invoice number that doesn't exist, a site Ecorp doesn't have, and an agreement that was never made. **How we decided the right answers:** every page that any method ranked highly, plus the pages of each question's target document, was graded 0, 1 or 2 by reviewers who didn't know which method had surfaced it. That's 691 judgments over 222 pages. Pages that no method surfaced count as not answering. This is the *pooling* approach search evaluations have long used. As in Ragextract, the LLM reads six pages per question, so a page ranked below sixth is a page it never sees. **The setup:** - **RAG:** pages embedded with Voyage's `voyage-multimodal-3.5` at 1,240 px, which is what Ragextract runs in production. - **JAG:** TypeSafe's Jev decision model (`typesafe/jev-1.14`), called through OpenRouter. ## Results <div style="overflow-x:auto"><svg viewBox="0 0 640 316" width="100%" role="img" aria-label="Headline results: JAG and JAG rerank score higher than RAG on every measure" style="min-width:520px;display:block;font-family:inherit"><rect x="215" y="8" width="12" height="12" rx="2" style="fill:var(--ink-faint)"/><text x="233" y="18" text-anchor="start" style="fill:var(--ink-muted);font-size:13px">RAG</text><rect x="276.6" y="8" width="12" height="12" rx="2" style="fill:var(--accent)"/><text x="294.6" y="18" text-anchor="start" style="fill:var(--ink-muted);font-size:13px">JAG</text><rect x="338.20000000000005" y="8" width="12" height="12" rx="2" style="fill:var(--spark-ink)"/><text x="356.2" y="18" text-anchor="start" style="fill:var(--ink-muted);font-size:13px">JAG rerank</text><line x1="215" y1="34" x2="215" y2="290" style="stroke:var(--line)"/><text x="215" y="306" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.0</text><line x1="407.5" y1="34" x2="407.5" y2="290" style="stroke:var(--line)"/><text x="407.5" y="306" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.5</text><line x1="600" y1="34" x2="600" y2="290" style="stroke:var(--line)"/><text x="600" y="306" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">1.0</text><text x="203" y="70" text-anchor="end" style="fill:var(--ink);font-size:13px">First page is an answer</text><rect x="215" y="44" width="231" height="12" rx="2" style="fill:var(--ink-faint)"/><text x="452" y="54" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.60</text><rect x="215" y="60" width="365.8" height="12" rx="2" style="fill:var(--accent)"/><text x="586.8" y="70" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.95</text><rect x="215" y="76" width="365.8" height="12" rx="2" style="fill:var(--spark-ink)"/><text x="586.8" y="86" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.95</text><text x="203" y="134" text-anchor="end" style="fill:var(--ink);font-size:13px">An answer in the top 6</text><rect x="215" y="108" width="365.8" height="12" rx="2" style="fill:var(--ink-faint)"/><text x="586.8" y="118" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.95</text><rect x="215" y="124" width="385" height="12" rx="2" style="fill:var(--accent)"/><text x="606" y="134" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">1.00</text><rect x="215" y="140" width="385" height="12" rx="2" style="fill:var(--spark-ink)"/><text x="606" y="150" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">1.00</text><text x="203" y="198" text-anchor="end" style="fill:var(--ink);font-size:13px">Answer pages in the top 6</text><rect x="215" y="172" width="338.8" height="12" rx="2" style="fill:var(--ink-faint)"/><text x="559.8" y="182" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.88</text><rect x="215" y="188" width="361.9" height="12" rx="2" style="fill:var(--accent)"/><text x="582.9" y="198" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.94</text><rect x="215" y="204" width="365.8" height="12" rx="2" style="fill:var(--spark-ink)"/><text x="586.8" y="214" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.95</text><text x="203" y="262" text-anchor="end" style="fill:var(--ink);font-size:13px">Tells answerable from absent</text><rect x="215" y="236" width="296.4" height="12" rx="2" style="fill:var(--ink-faint)"/><text x="517.5" y="246" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.77</text><rect x="215" y="252" width="373.4" height="12" rx="2" style="fill:var(--accent)"/><text x="594.5" y="262" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.97</text><rect x="215" y="268" width="377.3" height="12" rx="2" style="fill:var(--spark-ink)"/><text x="598.3" y="278" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">0.98</text></svg></div> | 20 answerable questions, 300 pages | RAG | JAG | JAG rerank | |---|--:|--:|--:| | First page shown is an answer | 12 of 20 | 19 of 20 | 19 of 20 | | At least one answer in the top 6 | 19 of 20 | 20 of 20 | 20 of 20 | | Answer pages in the top 6 (recall@6) | 0.88 | 0.94 | 0.95 | | Mean reciprocal rank | 0.72 | 0.96 | 0.96 | | Tells answerable from absent (AUC) | 0.77 | 0.97 | 0.98 | Both JAG shapes put an answer page first on 19 of 20 questions; RAG manages 12. On one question, RAG never showed the LLM an answer page at all. The chart below shows the eight questions where the methods disagree. The other twelve tie, with an answer page first for all three. <div style="overflow-x:auto"><svg viewBox="0 0 640 314" width="100%" role="img" aria-label="Rank of the first answer page: RAG ranks it as low as 12th, JAG first on all but one" style="min-width:520px;display:block;font-family:inherit"><rect x="250" y="8" width="12" height="12" rx="2" style="fill:var(--ink-faint)"/><text x="268" y="18" text-anchor="start" style="fill:var(--ink-muted);font-size:13px">RAG</text><rect x="311.6" y="8" width="12" height="12" rx="2" style="fill:var(--accent)"/><text x="329.6" y="18" text-anchor="start" style="fill:var(--ink-muted);font-size:13px">JAG and JAG rerank</text><line x1="250" y1="34" x2="250" y2="268" style="stroke:var(--line)"/><text x="250" y="286" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">#1</text><line x1="310" y1="34" x2="310" y2="268" style="stroke:var(--line)"/><text x="310" y="286" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">#3</text><line x1="400" y1="34" x2="400" y2="268" style="stroke:var(--line)"/><text x="400" y="286" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">#6</text><line x1="490" y1="34" x2="490" y2="268" style="stroke:var(--line)"/><text x="490" y="286" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">#9</text><line x1="580" y1="34" x2="580" y2="268" style="stroke:var(--line)"/><text x="580" y="286" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">#12</text><line x1="415" y1="34" x2="415" y2="268" style="stroke:var(--ink-muted);stroke-dasharray:4 4"/><text x="421" y="304" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">pages after #6 are never read</text><text x="236" y="56" text-anchor="end" style="fill:var(--ink);font-size:13px">Invoice AEC-2025-104276 total</text><line x1="250" y1="52" x2="580" y2="52" style="stroke:var(--line);stroke-width:3"/><circle cx="580" cy="52" r="6" style="fill:var(--ink-faint)"/><circle cx="250" cy="52" r="6" style="fill:var(--accent)"/><text x="236" y="84" text-anchor="end" style="fill:var(--ink);font-size:13px">Invoice AEC-2024-107812 period</text><line x1="250" y1="80" x2="400" y2="80" style="stroke:var(--line);stroke-width:3"/><circle cx="400" cy="80" r="6" style="fill:var(--ink-faint)"/><circle cx="250" cy="80" r="6" style="fill:var(--accent)"/><text x="236" y="112" text-anchor="end" style="fill:var(--ink);font-size:13px">Photos, Data Centre South visit</text><line x1="340" y1="108" x2="400" y2="108" style="stroke:var(--line);stroke-width:3"/><circle cx="400" cy="108" r="6" style="fill:var(--ink-faint)"/><circle cx="340" cy="108" r="6" style="fill:var(--accent)"/><text x="236" y="140" text-anchor="end" style="fill:var(--ink);font-size:13px">Attendees, 13 Nov 2025 meeting</text><line x1="250" y1="136" x2="370" y2="136" style="stroke:var(--line);stroke-width:3"/><circle cx="370" cy="136" r="6" style="fill:var(--ink-faint)"/><circle cx="250" cy="136" r="6" style="fill:var(--accent)"/><text x="236" y="168" text-anchor="end" style="fill:var(--ink);font-size:13px">Site visit recommendation</text><line x1="250" y1="164" x2="370" y2="164" style="stroke:var(--line);stroke-width:3"/><circle cx="370" cy="164" r="6" style="fill:var(--ink-faint)"/><circle cx="250" cy="164" r="6" style="fill:var(--accent)"/><text x="236" y="196" text-anchor="end" style="fill:var(--ink);font-size:13px">Invoice AEC-2024-105750 due date</text><line x1="250" y1="192" x2="280" y2="192" style="stroke:var(--line);stroke-width:3"/><circle cx="280" cy="192" r="6" style="fill:var(--ink-faint)"/><circle cx="250" cy="192" r="6" style="fill:var(--accent)"/><text x="236" y="224" text-anchor="end" style="fill:var(--ink);font-size:13px">Day rate from 1 Sep 2024</text><line x1="250" y1="220" x2="280" y2="220" style="stroke:var(--line);stroke-width:3"/><circle cx="280" cy="220" r="6" style="fill:var(--ink-faint)"/><circle cx="250" cy="220" r="6" style="fill:var(--accent)"/><text x="236" y="252" text-anchor="end" style="fill:var(--ink);font-size:13px">Who signed the supply agreement</text><line x1="250" y1="248" x2="280" y2="248" style="stroke:var(--line);stroke-width:3"/><circle cx="280" cy="248" r="6" style="fill:var(--ink-faint)"/><circle cx="250" cy="248" r="6" style="fill:var(--accent)"/></svg></div> The worst miss is the most instructive. *"What is the total amount due on invoice AEC-2025-104276?"* RAG ranks that invoice's page 12th. The bundle holds 20 Acme invoices with identical layouts. To an image embedding they all look alike, and a ten-character invoice number barely moves the vector. A judge reads the number. JAG ranks the right page first. | Recall@6 by kind of question | RAG | JAG | JAG rerank | |---|--:|--:|--:| | Exact identifiers and dates (7) | 0.79 | 0.93 | 0.93 | | Values in tables (6) | 1.00 | 1.00 | 1.00 | | Handwriting, signatures, photos (4) | 1.00 | 1.00 | 1.00 | | Findings and reasons (3) | 0.69 | 0.80 | 0.82 | Image embeddings hold up well on tables and visual content. The gap opens on exact identifiers and on questions whose answers are spread across several pages. ## Knowing when the answer isn't there For an extraction product, this is the result that matters most. When a column has no answer in a document, the right output is an empty cell. RAG can't tell you that: it always returns six pages, and the LLM is left to work out that none of them helps. Often it tries anyway. Could a cut-off on the retrieval score do the job? Here is each question's best score, with answerable questions in blue and absent ones in red: <div style="overflow-x:auto"><svg viewBox="0 0 640 256" width="100%" role="img" aria-label="Best-page scores: RAG's answerable and absent questions overlap; JAG separates all but two" style="min-width:520px;display:block;font-family:inherit"><text x="0" y="18" text-anchor="start" style="fill:var(--ink);font-size:13px;font-weight:600">RAG: embedding similarity of the best page</text><text x="136" y="52" text-anchor="end" style="fill:var(--ink-muted);font-size:13px">Answerable (20)</text><text x="136" y="84" text-anchor="end" style="fill:var(--ink-muted);font-size:13px">Absent (10)</text><line x1="150" y1="32" x2="150" y2="96" style="stroke:var(--line)"/><text x="150" y="112" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.30</text><line x1="281.4" y1="32" x2="281.4" y2="96" style="stroke:var(--line)"/><text x="281.4" y="112" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.40</text><line x1="412.9" y1="32" x2="412.9" y2="96" style="stroke:var(--line)"/><text x="412.9" y="112" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.50</text><line x1="544.3" y1="32" x2="544.3" y2="96" style="stroke:var(--line)"/><text x="544.3" y="112" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.60</text><circle cx="466.7" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="491.7" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="464.1" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="536.4" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="385.3" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="410.2" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="583.7" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="527.2" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="508.8" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="302.5" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="485.1" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="343.2" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="429.9" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="498.3" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="491.7" cy="44" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="411.5" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="319.5" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="531.1" cy="44" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="498.3" cy="44" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="541.7" cy="48" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="345.8" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="410.2" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="340.6" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="470.7" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="529.8" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="194.7" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="464.1" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="249.9" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="284.1" cy="80" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="348.5" cy="76" r="5" style="fill:var(--danger);fill-opacity:0.75"/><text x="0" y="150" text-anchor="start" style="fill:var(--ink);font-size:13px;font-weight:600">JAG: the judge's probability for the best page</text><text x="136" y="184" text-anchor="end" style="fill:var(--ink-muted);font-size:13px">Answerable (20)</text><text x="136" y="216" text-anchor="end" style="fill:var(--ink-muted);font-size:13px">Absent (10)</text><line x1="150" y1="164" x2="150" y2="228" style="stroke:var(--line)"/><text x="150" y="244" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.00</text><line x1="265" y1="164" x2="265" y2="228" style="stroke:var(--line)"/><text x="265" y="244" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.25</text><line x1="380" y1="164" x2="380" y2="228" style="stroke:var(--line)"/><text x="380" y="244" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.50</text><line x1="495" y1="164" x2="495" y2="228" style="stroke:var(--line)"/><text x="495" y="244" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">0.75</text><line x1="610" y1="164" x2="610" y2="228" style="stroke:var(--line)"/><text x="610" y="244" text-anchor="middle" style="fill:var(--ink-muted);font-size:12px">1.00</text><circle cx="605.4" cy="180" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="176" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="184" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="172" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="188" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="170" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="190" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="170" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="190" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="170" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="190" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="170" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="190" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="170" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="190" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="605.4" cy="170" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="596.2" cy="180" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="596.2" cy="176" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="596.2" cy="184" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="467.4" cy="180" r="5" style="fill:var(--accent);fill-opacity:0.75"/><circle cx="228.2" cy="212" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="177.6" cy="212" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="591.6" cy="212" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="182.2" cy="208" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="168.4" cy="212" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="159.2" cy="212" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="596.2" cy="208" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="163.8" cy="208" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="173" cy="212" r="5" style="fill:var(--danger);fill-opacity:0.75"/><circle cx="173" cy="208" r="5" style="fill:var(--danger);fill-opacity:0.75"/><line x1="288" y1="164" x2="288" y2="228" style="stroke:var(--ink-muted);stroke-dasharray:4 4"/><text x="294" y="164" text-anchor="start" style="fill:var(--ink-muted);font-size:12px">cut-off 0.3</text></svg></div> The embedding scores for answerable and absent questions overlap almost completely. The invoice number we made up, AEC-2025-101234, scores **0.589**, higher than 16 of the 20 real answers. A similarity score measures how alike two things look, not whether a question is answered. The judge's probabilities separate cleanly, with two exceptions: | 10 absent questions | Cut-off | Answerable kept | Absent caught | |---|--:|--:|--:| | RAG similarity, at its best cut-off | 0.416 | 20 of 20 | 3 of 10 | | JAG | 0.3 | 20 of 20 | 7 of 10 | | JAG rerank | 0.3 | 20 of 20 | 8 of 10 | The two that fooled the judge are worth understanding: - **"When can the Metering Services Agreement be terminated?"** There is no Metering Services Agreement. But the *Electricity Supply Agreement* has a termination clause, and the judge scored it 0.97. - **"What settlement sum did the parties agree?"** No settlement was ever reached. But an email proposes clearing a balance in three instalments, which scored 0.96. A judge is good at "not here", and weaker at "something very like it is here". For a product, that means a cell can say *not found* with confidence. When a near-miss does score high, the LLM still reads the page and can still decline to answer. ## Cost and speed | This 300-page bundle, 30 questions | RAG | JAG | JAG rerank | |---|--:|--:|--:| | Embedding every page (once per document) | $0.196 | — | $0.196 | | Judging pages | — | $0.080 (all 300) | $0.023 (160 shortlisted) | | Re-run when a column is added later | free | re-reads every page | the new column's 15 pages | A few things stand out: - **Judging is cheap.** A page image costs about 3,130 input tokens, and each extra question adds about 107. One request per page answers every column at once. - **Full JAG can cost less than embedding,** as it did here. But embeddings are paid once and serve any future question, while a judgement only answers the questions it was asked. - **It's fast.** A judgement came back in about 0.37 seconds (median), and pages run in parallel. - **These are retrieval costs only.** The LLM that writes the answer is the same in all three. ## Conclusions ### What does this mean for Ragextract? If we add a judge to Ragextract, it will be **JAG rerank**: RAG shortlists, and the judge decides. #### Pros - **The right page comes first more often.** An answer page was ranked first on 19 of 20 questions, against RAG's 12. - **A trustworthy *not found*.** It caught 8 of the 10 absent questions without losing an answerable one. RAG's similarity scores caught 3. - **It's the best of the three.** It matched or beat full JAG on every measure in the results table. - **It's cheap and fast.** The judging bill is capped at the shortlist: $0.023 for 30 questions here. A judgement takes about 0.37 seconds (median), and a column added later only needs its own 15 pages judged. #### Cons - **We won't put customer documents through Jev yet.** Jev performed well here. But as we read TypeSafe's [Master Customer Agreement](https://typesafe.ai/legal/mca) (last updated 23 September 2026), there's a contractual promise not to train models on customer data without consent (section 4.1), which is good. The same agreement also lets TypeSafe keep information *derived* from what customers send. It calls this Telemetry, which includes "learnings related to Customer's use of the Services", and may use it "without restriction, including to improve the Services or TypeSafe's other products and services" (sections 4.1 and 4.3). And according to TypeSafe's [legal page](https://docs.typesafe.ai/legal), zero data retention is offered to enterprise customers through its sales team. Ragextract's users upload contracts, invoices, medical letters and case files. For data like that, those terms aren't something we'll accept without clarity in writing. So the next step is a different decision model, or terms we can sign up to. The approach doesn't depend on Jev: any model that returns a calibrated yes/no on a page image can be the judge. - **A near miss can still fool the judge.** Two of the ten absent questions scored 0.96 or higher, because something very like the answer was in the bundle. The LLM still reads those pages and can still decline to answer. - **It adds to the bill rather than replacing it.** Embedding every page still costs what it did, and judging comes on top. #### What doesn't change - **Search.** The vector store stays, so questions nobody wrote as a column still work. - **How answers are written.** The LLM still reads six pages per question and writes the answer, as it does today. - **What happens to your documents.** Nothing here has shipped. Ragextract retrieves pages exactly as described at the top of this post, and no customer document is sent to a decision model. ### Is this the end of RAG? Maybe. It depends on whether you know your questions before the documents arrive. If you do, as with an extraction table, a review checklist or a due diligence questionnaire, a judge did better in our test: it put the right page first more often, and it could tell when there was nothing to find. Full JAG did that with no embeddings and no vector store at all. If you don't, as in a chatbot or a search box, you still need an index to search, and that means RAG. The sums also change with scale: an embedding is paid for once and serves any future question, while judging grows with every page and every question. For many jobs the answer may be both, which is where we've landed for Ragextract. One synthetic bundle can't settle it for you. That depends on your documents: how many pages there are, how often the questions change, and what a missed answer or an invented one costs you. **If you'd like to know whether JAG is right for your documents, we can help you evaluate it.** We'll test it on a sample of your own documents and questions, measured the way we did here, and tell you plainly whether JAG, RAG or both suits them. [Get in touch](/contact-us) or email [hello@subworkflow.ai](mailto:hello@subworkflow.ai). *All figures in this post come from a synthetic test bundle generated for the purpose. No customer documents were used.*