The question “which AI analyzes PDFs best?” It seems simple, but it is often answered with the wrong criteria. Accepting a PDF upload does not mean seeing everything in it, locating the correct evidence or resisting the temptation to fill in a gap with a plausible answer. To usefully compare ChatGPT, Claude, Gemini, and NotebookLM, the test needs to separate text reading, visual reading, OCR, table reconstruction, page localization, and hallucination control.
On August 17, 2026, the documentation for the four platforms already shows differences that change the result even before any questions are asked. Therefore, it is not correct to declare a universal champion without running the same files, prompts and criteria under the same conditions. The most reliable path is to use a reproducible benchmark, record the test plan, model and date, and publish the raw responses as well, not just a final score.
The first cut is understanding that not all PDFs are the same problem
A PDF with digital text allows a tool to extract characters and search for passages without necessarily “looking” at the page. A scanned PDF can just be a collection of images. Tables, graphs, diagrams, stamps and notes can also carry information that disappears if the system works only with extracted text.
This distinction explains why an AI can get the answer to a question on one paragraph perfectly right and fail on the graph on the next page. It's not a contradiction: it could be a difference in the way the file is ingested. A benchmark that mixes everything in a single note, without separating text and vision, hides precisely the type of flaw that the
reader needs to know.
What the current documentation already requires you to control in the test
ChatGPT: the plan and way of attaching the PDF changes what can be seen
The OpenAI documentation states that ChatGPT accepts document uploads and that text files and documents have a limit of 2 million tokens per file, in addition to the hard limit of 512 MB per file. The most important point for this comparison, however, is visual: visual retrieval of elements embedded in PDFs is documented as a feature of ChatGPT Enterprise.
Other than this condition, the documentation says that document processing is text-based and that embedded images are discarded. There is also a difference in flow within Enterprise itself: PDFs used as project files or GPT knowledge are processed by textual retrieval, while PDFs sent by the user during the conversation can use visual retrieval. Therefore, a serious test needs to record not only “ChatGPT” but also the upload plan and path.
Claude: Short and long PDFs come in different ways
Anthropic documents that Claude can analyze text and visuals from PDFs up to 100 pages long. Between 101 and 1000 pages, processing becomes textual only. The platform also advises that, when requesting location, the user refers to the page number shown by the PDF viewer, and not to the number printed inside the document.
This creates two essential controls. The first is to keep the visual PDF below 100 pages when the proof depends on a graph, image or scanned page. The second is to define in advance which number is valid in the template. Without this, an answer could be semantically correct and still appear wrong because the document calls “page 1” something that the PDF file considers page 7.
Gemini: you need to separate the application from the capabilities documented in the API
In Gemini Apps, Google documents document upload and analysis, with up to 10 files in the same command when the feature is available and a limit of 100 MB for each accepted file that is not video. The help itself warns that very large files may produce responses with fewer correlations or details when the content exceeds the available context window.
In the Gemini API, the technical documentation is more explicit about multimodal PDF: Google describes native vision capable of analyzing text, images, diagrams, graphs and tables in documents of up to 1000 pages. This is an important capability, but it does not authorize treating “Gemini Apps” and “Gemini API” as if they were exactly the same product. The benchmark must choose a surface and stay on it from start to finish.
NotebookLM: traceability is part of the product, but does not replace the template
NotebookLM works from the fonts selected on the notebook. Google's documentation states that the chat uses excerpts, text and images from the sources themselves as quotes and allows you to navigate from the quote to the original context. It also records a limit of up to 500,000 words per source or 200 MB for local uploads, with no declared page limit, and informs that copy-protected PDFs may fail to import.
This architecture makes traceability especially easy to measure: it's not enough to ask if the answer is right; it is possible to check whether the quote actually supports the sentence. Still, citation is not synonymous with accuracy. An answer may point to a real snippet and misinterpret that snippet. In the benchmark, “correct answer” and “correct evidence” need to be scored separately.
Why a simple ranking often rewards the wrong thing
Showcase comparisons often measure whether the tool accepted the file, generated a convincing summary, and answered some obvious questions. This promotes fluency. The problem is that the user who consults a financial report, a notice, a paper or a technical manual generally needs something else: verifiable accuracy.
A tool can sound excellent and still change a table line, confuse an absolute value with a percentage, indicate the wrong page or invent missing data. Another may be more cautious, respond less and obtain a better success rate. If the goal is reliability, a conscious omission may be preferable to a complete and incorrect answer.
How to create a benchmark PDF that has a real answer
The best test file is synthetic. This avoids copyrights, allows you to publish the complete material and gives you control over each response. Instead of using a real report and discussing later which interpretation was correct, create a fictitious document with numbers, text, tables and graphs designed specifically for the test.
An example can be 24 pages long. The first pages feature normal digital text, with facts distributed in different sections. Then, tables with similar headers and similar values appear.. Then, graphs appear that contain information not repeated in the text. The final pages reproduce the same fictional universe as scans, with small rotations, slight noise, and reduced contrast to test OCR without turning the file into an impossible trap.
Content should include deliberately similar values. For example, a margin of 23.7%, an improvement of 2.8 percentage points, and revenue of 1,642 units may appear in different contexts. This forces the system to distinguish between unit, period and metric, rather than simply finding the number that appears to match the question.
It is also worth inserting questions whose correct answer is “not included in the document”. This is one of the most informative tests, because it measures whether the tool recognizes the absence of evidence or fabricates a likely continuation.
Questions need to measure different skills
Factual extraction
Ask questions whose answer is literally in the digital text and ask for the value, unit and period. The score should drop if the AI gets the number right but gets the unit or context wrong. This block measures the most basic capability and serves as a control: if the tool fails here, any more sophisticated analysis needs to be interpreted with caution.
Page and excerpt location
Ask for the answer accompanied by the PDF file page and the section title. Define in the statement that “page” means the position shown in the viewer, starting from the first physical page of the file. This reduces the ambiguity between printed pagination and PDF pagination.
The test must verify that the location is accurate, not just close. A tool that responds correctly but points to the wrong page makes auditing difficult and increases the cost of human review.
Table reconstruction
Choose a table in which rows and columns have similar names. Order only a subset, such as three indicators in two periods, and determine a fixed order for the output. The template must check cell by cell.
The most dangerous flaw is not an incomplete table; is the visually organized table with displaced cells. Therefore, the scoreboard must distinguish omission of value and wrong association between line, column and number.
Graph reading
Include at least one graphic where the conclusion cannot be retrieved from the surrounding text. Ask which series achieves the highest value, in what period this occurs and what visual relationship exists between two series. Therefore, a tool that only extracts text cannot pass the test by coincidence.
This block is especially useful for revealing differences in plan and processing mode. If the system does not receive the images embedded in the PDF, the result must be marked as an input limitation, not as a simple reasoning error.
OCR on scanned page
Create a page without a searchable text layer and confirm this before testing. The question must require the reading of a sentence and a number present only in the image. Make a second version with slight tilt or noise to measure robustness.
The template must consider characters, accents, decimal separators and negative signs.
An “almost certain” reading may be useless in legal or financial documents, so the criteria needs to be more rigorous than in a narrative summary.
Hallucination control
Include plausible questions about information that does not exist in the file. The ideal response is to acknowledge the absence and, when possible, indicate which related elements were found. Inventing a value, a date, or a conclusion should receive the highest benchmark penalty.
This test prevents a tool from earning points just for always responding. In document analysis, refusing a conclusion without evidence may be a sign of reliability, not weakness.
A simple rubric makes the result auditable
Each question can receive 0, 1 or 2 points. Zero means wrong, invented or incompatible with the evidence answer. A dot means a partially correct answer, with omission or imprecise location. Two points mean correct answer, complete within the request and traceable to the answer key.
To prevent easy questions from dominating the grade, the final result can be weighted. A reasonable distribution for this type of test is 40% for factual accuracy, 20% for page and excerpt traceability, 15% for tables, 15% for visual reading and OCR, and 10% for hallucination control. The weights are an editorial choice of the benchmark and must be published together with the result.
It is also important to repeat each question. Three independent runs per tool help reveal unstable responses. The scoreboard can use the average of the three rounds and record, separately, how many times the answer changed from correct to incorrect without changing the prompt.
Test conditions need to be stricter than the scoreboard
Each tool must start in a new conversation or notebook, with no history related to the benchmark. The same file must be sent directly by the surface being evaluated. Web search, connectors, memory of previous conversations, and external tools should remain disabled when responding exclusively from the PDF.
The prompt must be identical whenever the interfaces allow it. It's not worth correcting the tool after a bad response and comparing this second attempt with the competitors' first attempt. If there is a repair round, it should be a separate step and applied to all four.
Record date, plan, model name when visible, platform used, upload path and any relevant settings. AI products change quickly; Without this metadata, a result published today may be impossible to reproduce a few months later.
What to publish for the benchmark to be truly reproducible
The public package must contain the original PDF, the answer key, the set of questions, the rubric, the weight of each criterion and the hash of the tested file. You should also save the raw responses from the tools, without editorial rewriting, in addition to the score calculated from them.
When a tool is unable to process a component due to a documented limitation of the chosen plan or flow, this should appear as a result. Don't replace the PDF with single images just for a competitor, unless there is a second category explicitly called “best result with manual preparation”. Mixing conditions turns the benchmark into a demonstration, not a comparison.
What can you conclude before running the scoreboard
The documentation already allows us to formulate hypotheses, but not to crown a winner. In visual PDFs, a ChatGPT outside of the Enterprise flow with visual retrieval starts from a different condition because the documentation says that embedded images are discarded in textual processing. Claude documents visual analysis up to 100 pages and textual processing in larger PDFs. Gemini has native PDF view technical documentation in the API, while Gemini Apps have their own limits and context that need to be tested separately. NotebookLM provides a citation and navigation layer for the source that makes it easy to audit the response.
These differences suggest that there will be winners per task, not necessarily one dominant tool at all. A system may lead in traceability and lag behind in reconstructing a scanned table; another may interpret graphics very well and be less consistent in locating pages. The benchmark needs to preserve these dimensions rather than hiding them in a single note.
How to choose an AI for your PDF without waiting for the full benchmark
Start by identifying the type of document. If everything is digital text, ask five difficult and verifiable questions in distant parts of the file. If there are graphs, scanned pages or visual tables, first confirm that the plan and flow used actually renders these elements.
Then, always ask for evidence. Request a supporting page, section and excerpt, and open the PDF to check it out. Ask at least one question for which you already know the answer and another about something that is not included in the document. This small test does not replace a benchmark, but it quickly detects tools that respond with greater confidence than the evidence allows.
In the end, the best AI for PDF is not the one that produces the most elegant summary. It is the one that best matches your type of document, makes detectable errors and shows the origin of the response. Without this, “accepts PDF” is an interface feature, not a measure of reliability.
