How Alevr reads your files

A modern assistant does not do PDF → text → model. It does PDF → (text layer ∥ page images) → model, in parallel, and the model sees both.

The text carries exact strings and reading order. The pixels carry layout, diagrams, handwriting, stamps, table rules, and everything else a text extractor throws away. Retrieval is a scaling strategy applied on top when a document is too large to show whole — it is not the primary way a file gets read. And the heavy lifting that is genuinely deterministic — parsing, decoding, rendering — stays in ordinary software, because a parser is cheaper, faster and more reliable than a model at work that has a correct answer.

Anthropic documents its PDF pipeline in two sentences that settle the central question, so they are worth quoting exactly rather than paraphrasing:

Google documents the same design for Gemini — every page is rasterised to a screenshot and OCR’d, which is precisely why scanned PDFs work there without any preparation PublishedGemini document processing. OpenAI’s Responses API accepts a PDF as a first-class input_file part carrying a render-quality detail knob, which only means something if pages are being rendered Publishedopenai-openapi, InputFileContentParam.

Three vendors, three independent specifications, one shape: dual channel. Text and pixels, per page, together. If you remember one thing from this page, that is the thing.

The cost of the second channel is documented too. The visual path runs roughly seven times the token cost of text alone — on the order of 1,500–3,000 text tokens per page plus a full image’s worth of tokens per page PublishedPDF support — “Estimate your costs”. Bedrock exposes both modes side by side so a caller can choose, which is the clearest public evidence that these are two genuinely different architectures rather than one pipeline with a flag.

The walkthrough

A 42-page PDF is uploaded, and the question is “what does page 37 say about the database architecture?”

So is it PDF → text → model?

No — and the failure of a system built that way is not subtle. A scanned page yields the empty string. A diagram yields its labels with the arrows removed. A two-column paper yields two columns interleaved line by line. Microsoft Copilot is the public cautionary case: a text-extraction pipeline with a character ceiling that fails outright on image-only PDFs PublishedCopilot Studio document handling.

An image takes a different route, and one step in it surprises almost everyone.

1. 01Decode — Bytes to a pixel array. The format is sniffed from magic bytes, not the file name.
2. 02Resize — Providers downscale before inference. Anthropic documents a maximum of 8000×8000 px and roughly 1,590 tokens for a 1.15 megapixel image; Gemini documents 258 tokens per tile. This is why a 4 % crop of a 4000 px screenshot is useless: it is 160 px before the provider shrinks it again.
3. 03Patchify — The image is cut into fixed-size patches — typically 14×14 or 16×16 pixels — each linearly projected into an embedding.
4. 04Vision encoder — A ViT-family transformer processes the patch sequence. Self-attention across patches is what encodes spatial relationship: this patch is left of that one, this arrow starts here and ends there.
5. 05Projection — Visual embeddings are mapped into the language model’s embedding space, becoming tokens it attends over alongside text tokens.
6. 06Joint reasoning — From the model’s point of view there is one sequence. It does not call a vision module; the picture is already in its context as vectors.

PublishedThe token arithmetic is documented — Anthropic vision and Gemini. InferredSteps 3 to 5 are the standard published vision-language architecture — CLIP, Flamingo, LLaVA, Qwen-VL — and the vendors’ token counts are consistent with fixed-size patching. Not publicWhether the frontier models implement exactly this is not public.

Why “what button is in the top right?” works

Because position survives. Patches enter the sequence with positional encodings, so “top right” is recoverable from the representation. OCR would have returned the button’s label and thrown away where it was.

Four things that get conflated

Run OCR on an architecture diagram and you get Database API Frontend. The arrows — which are the architecture — are not characters, so they do not survive. That is the whole reason the dual-channel design exists.

“Convert it to text” is not a neutral operation. Every format loses something specific, and knowing which thing is the difference between a pipeline that degrades and one that lies.

Vendors are explicit about the limits here. Anthropic’s document block accepts only application/pdf and text/plain; .docx and .xlsx are rejected outright and must be converted first PublishedFiles API. Their own recommendation for a Word document containing images is to convert it to PDF and use the PDF path — precisely so the images survive.

Spreadsheets get a different path entirely. Published.xlsx and .csv go through code execution rather than document blocks. That is a deliberate architectural choice, and the right one: a spreadsheet question is usually a computation — “what is the Q3 total?” — and putting 40,000 cells into a context window to make a model add them up is both expensive and unreliable. Run pandas instead.

Take a page containing one diagram. What the parser gets and what the model needs are not the same object:

Three nouns. The arrows are gone, so the direction of data flow — the entire content of the diagram — is gone with them. A model given only that text cannot answer “does the frontend talk to the database directly?”, and, worse, will usually answer anyway.

A strong system keeps both as parallel representations of the same page, joined by page number. A semantic layer — blocks with types, text, heading path and bounding boxes — is what exact quotation, search and citation run on. A visual layer — the rendered page — is what layout, arrows, charts, handwriting and stamps survive in. At answer time you send both, for the pages that matter.

The bounding boxes are what let you go further than “send the page”. If block seven on page fifteen is an image at a known rectangle, you can crop just that region and send it magnified — far cheaper than the whole page and far more legible than a downsampled one. In AlevrThat is what Alevr’s image inspection tool does: a region given in percentages, cropped and scaled up to at least 768 px on its short edge before the model ever sees it.

A 500-page PDF cannot go in every prompt. At roughly 2,000 text tokens a page that is a million text tokens, and with the visual channel considerably more. There are five honest approaches and they are not interchangeable.

The common mistake is jumping straight to B for a twelve-page PDF. That is infrastructure solving a problem you do not have — and it loses information relative to simply sending the document.

Threshold on size instead. If it fits the window, send it whole. If it does not, use hybrid retrieval over blocks with page and section filters, and rerank. If the question is “summarise this 500-page report”, retrieval cannot answer it at all — the answer is the whole document — so that is where hierarchical summaries earn their keep. Add page-image retrieval only when the documents are visually dense enough to need it.

“Context” sounds like one thing a model does. It is eleven things, and only four of them involve a model at all. Take “compare the authentication architecture described in sections 3 and 7.”

Two rows deserve emphasis. An explicit locator short-circuits everything — “page 37” is a filter, and running a semantic search for those words is strictly worse than applying it. And the last row is not optional: checking citations against what was actually retrieved, after generation, is how you stop hallucinated page numbers reaching a reader.

Generic lists of mistakes are cheap, so this one is grounded: every item below was a real defect in Alevr, found during the audit, with the symptom it produced in front of real people. All of them are fixed.

Architectural

1. Analysing the file at upload, before a question exists. An extractor ran on arrival and its verdict became permanent, so a misjudged PDF was “couldn’t read this file” forever. Read when asked, not when stored.
2. Treating extraction as a gate rather than an optimisation. If extraction fails the file should still be readable by other means — the bytes to the model, the pages as images, a code tool. It was the gate.
3. One reader, no ladder. The ladder existed but was unreachable: the code returned on failed before the rung that could have rescued the file.
4. Confusing retrieval with reading. Running retrieval over a single document the user is looking at is the wrong tool — there is no irrelevant part to filter out. Four retrieved passages stood in for a whole report.

Extraction

5. An OCR fallback that could never run. Worse than not having one: pdf.js transfers its input buffer — measured, 16,978 bytes to 0 — so the OCR rung downstream received an empty file on every single document. Silent, and indistinguishable from “this scan has no text”.
6. Trusting the declared MIME type. Browsers send application/octet-stream constantly. A PDF labelled that way skipped the raw-bytes path entirely.
7. The gate and the reader disagreeing. The upload gate judged MIME type while the router judged file extension, so a .sql file was accepted from macOS and refused from Linux — the same bytes.
8. Refusing formats we could already read. .docx, .xlsx and .pptx were rejected at upload with working extractors sitting behind the wall.
9. Indexing markup as prose. RTF had no extractor, fell through the plain-text arm, and filled the index with \fonttbl and \pard.
10. Not sanitising for the datastore. A single NUL byte failed the insert for an entire 200-page document, because PostgreSQL’s text type cannot hold 0x00. One bad byte, zero blocks stored.

Chunking and retrieval

11. Chunks too large, so retrieval returns noise — or too small, so it returns a sentence with no context. Roughly 200–500 tokens with overlap is a sane default.
12. Chunking across a table or a heading boundary, severing a row from its header.
13. No metadata. No page, no section, no file. A chunk you cannot locate is a chunk you cannot cite.
14. Confusing pages with chunks. A chunk is a retrieval unit; a page is a citation unit. They are not the same and you need both.
15. Embeddings where exact search belongs. Identifiers, error codes, function names: that is BM25 territory. A stemmer will not find NDA-4417.
16. No reranking. Vector top-k is a rough ordering, and a cross-encoder over the top fifty is the cheapest large quality win available.
17. Sending everything retrieved. Irrelevant context measurably degrades answers, and costs money to do it.

Interface and honesty

18. Passing judgement on a file before anyone asked. The composer said “couldn’t read this file” on the basis of a search index, while the actual answer depends on the model — which receives the PDF and reads a scan fine. Wrong in both directions.
19. Telling the model a file is unreadable while handing it the file. The system prompt said “no readable text — do not invent contents” on turns where the adapter had inlined the PDF two blocks earlier. The best case reported as the worst.
20. No size guard before base64 inlining. Base64 adds a third, so an oversized PDF failed the whole turn — the person lost their answer, not just their attachment.
21. Expecting the model to reconstruct what you destroyed. If you flatten a table, no amount of prompting recovers the cell relationships.

What each stage is for, when it runs, how it fails, and what it does instead of failing. Note how many of them are conditional: a well-built pipeline does the minimum a question requires, not the maximum a file permits.

1. 01

   Upload and validate

   Synchronous

   What

   Accept the bytes, enforce size and type policy, store them. Nothing is read.

   Why

   The only stage that must be synchronous, because a person is watching it.

   Fails when

   Quota, oversize, a type the policy refuses.

   Recovers by

   Rejecting with the actual type and the actual limit, not “upload failed”.

2. 02

   Type detection

   Synchronous

   What

   Magic bytes first, extension second, the browser’s declared type last.

   Why

   All three disagree in practice, and the declared one is the least reliable of them.

   Fails when

   Container formats. A ZIP is a .docx, an .xlsx, an .odt, an .epub, and a folder of holiday photos.

   Recovers by

   Disambiguating on the archive’s inner manifest.

3. 03

   Parse and extract structure

   Asynchronous

   What

   A format-specific parser producing typed blocks that each know where they came from.

   Why

   This is the stage where structure is preserved or lost permanently.

   Fails when

   Encrypted files, corrupt files, exotic font encodings, XFA forms.

   Recovers by

   The ladder — native parser, then a second parser, then OCR, then vision. No rung’s failure ends the file’s life.

4. 04

   Page rendering

   Conditional, async

   What

   Each page to an image at roughly 150 DPI. This is the visual channel, and the thumbnail, and the OCR front end.

   Why

   Everything a text extractor throws away is still in the pixels.

   Fails when

   No rasteriser available on the platform.

   Recovers by

   Degrading to text only — and saying so, rather than quietly answering anyway.

5. 05

   OCR

   Conditional, async

   What

   Text for pages that have no text layer, with per-word confidence and boxes.

   Why

   Scans. Only scans. OCR is strictly worse than embedded text wherever embedded text exists.

   Fails when

   No OCR engine installed; genuinely bad scans.

   Recovers by

   Falling through to the visual channel. A model that can see the page does not need OCR to answer about it.

6. 06

   Indexing

   Conditional, async

   What

   Chunk, embed, write to the lexical and vector indexes.

   Why

   Only for documents too large to show whole, or for searching across a corpus. Never for a ten-page attachment.

   Fails when

   The embedding provider is down; the text contains bytes the datastore cannot hold.

   Recovers by

   Retry with backoff, and degrade to lexical search rather than to nothing — BM25 works without embeddings.

7. 07

   Retrieval and rerank

   Synchronous

   What

   Locator filters, then BM25 and vectors in parallel, fused, then a cross-encoder over the top of the list.

   Why

   It sits in the request path, so it has a latency budget of a few hundred milliseconds.

   Fails when

   Nothing relevant is found.

   Recovers by

   Saying so explicitly. “No passage matched” has to reach the model, or it will fill the silence.

8. 08

   Context assembly

   Synchronous

   What

   Fit the budget: instructions, question, selected blocks with their locators, selected page images, history.

   Why

   The token budget is finite and shared. No model is involved in spending it.

   Fails when

   Budget overrun.

   Recovers by

   Dropping the lowest-ranked material first — and telling the model what was dropped. Silent truncation manufactures confident wrong answers.

9. 09

   Generation and citation validation

   Synchronous

   What

   Call the model, then check that every locator it cited resolves to something that was actually retrieved.

   Why

   This is the step that makes a hallucinated page number impossible to display.

   Fails when

   The answer cites a page that was never in the context.

   Recovers by

   Stripping or flagging the unresolvable citation before anyone reads it.

The problem with reducing a document to a string is that every locator, every type and every relationship is destroyed in one step, and nothing downstream can recover them. This is roughly the minimum that keeps the options open.

{
  "documentId": "doc_01H…",
  "source": {
    "fileName": "architecture-review.pdf",
    "mimeType": "application/pdf",
    "sha256": "…",
    "sizeBytes": 4718592
  },
  "metadata": {
    "title": "Platform Architecture Review",
    "pageCount": 42,
    "language": "en",
    "hasTextLayer": true,
    "parser": "pdfjs",
    "status": "ok"                  // ok | degraded | failed
  },

  "sections": [
    { "id": "sec_3", "title": "Authentication", "level": 1,
      "path": ["Architecture", "Authentication"],
      "startPage": 12, "endPage": 19 }
  ],

  "pages": [
    { "number": 15, "width": 612, "height": 792, "rotation": 0,
      "render": { "uri": "s3://…/page-015.jpg", "dpi": 150 },
      "hasTextLayer": true,
      "blockIds": ["blk_15_1", "blk_15_7"] }
  ],

  "blocks": [
    { "id": "blk_15_1",
      "ordinal": 271,               // reading order across the document
      "page": 15,
      "type": "heading",            // heading | paragraph | list_item | table
                                    // | code | caption | image | speaker_notes
      "text": "3.2 Token exchange",
      "headingPath": ["Architecture", "Authentication", "Token exchange"],
      "bbox": [72, 648, 468, 24],   // x, y, w, h in page units
      "confidence": 1.0,            // 1.0 embedded; below that, OCR, as measured
      "source": "embedded" },       // embedded | pdfjs | ocr | vision

    { "id": "blk_22_3",
      "ordinal": 402,
      "page": 22,
      "type": "table",
      "text": "Region | Revenue\nOntario | 4,200,000",   // for lexical search
      "table": {                                         // for computation
        "columns": ["Region", "Revenue"],
        "rows": [["Ontario", "4200000"], ["Quebec", "3100000"]]
      },
      "confidence": 1.0,
      "source": "embedded" }
  ],

  "chunks": [
    { "id": "chk_19",
      "blockIds": ["blk_15_1", "blk_15_2", "blk_15_3"],
      "text": "3.2 Token exchange\n\nThe service exchanges …",
      "pageStart": 15, "pageEnd": 15,
      "tokenCount": 380 }
  ]
}

Every field earns its place. ordinal restores reading order after any query has scrambled it. bbox is what lets you crop a figure or highlight a citation in a viewer. confidence and source keep OCR guesses distinguishable from verified text — merging them silently is how a reconstruction turns into a quotation. headingPath gives a chunk its context without re-reading the document. chunks.blockIds is the join that makes citation possible at all: chunk, to blocks, to page. And table.rows beside table.text means the same table can be searched lexically and computed over.

Ask “what database technology is used, and why?” of a document that says “we chose PostgreSQL for its JSONB support”. The first half of that question is a lexical match and the second half has no lexical overlap with the answer at all. That is the whole case for hybrid retrieval in one sentence.

• Lexical / BM25 catches PostgreSQL, pgvector, NDA-4417. Exact tokens, no embedding call, no semantic drift. Essential for identifiers, error codes, function names and part numbers.
• Embeddings catch the “why”, where the answer shares no words with the question.
• Reciprocal rank fusion merges the two. Parameter-free and robust, and it needs no score normalisation between two incomparable scales.
• Metadata filters run first. “In section 3”, “on page 37”, “in the appendix” are WHERE clauses. Applying them before the search is both cheaper and more correct.
• Rerank the top fifty down to the top eight with a cross-encoder. Bi-encoders embed the query and the document separately; a cross-encoder sees both together and is markedly better at relevance. This is usually the single biggest quality gain per unit of effort in the whole system.

Vector search alone is insufficient for a structural reason, not a tuning reason. Embeddings compress meaning into a few hundred dimensions, which is exactly the wrong operation for exact strings: NDA-4417 and NDA-4418 are near-identical vectors and completely different contracts. Cosine similarity also has no notion of sufficiency — the top five come back whether or not any of them answers the question, so a retrieval miss is indistinguishable from the document not containing the answer. The model cannot tell which happened unless you tell it.

Visual retrieval

“What does the diagram on page 24 show?” needs no retrieval at all. Page 24 is a locator: filter, take the render, send it. Running a semantic search there is a strictly worse way to answer an exact question, and it is the most common over-engineering mistake in document question-answering.

“Show me the diagram of the auth flow”, with no page given, is the case that needs real visual retrieval. Four strategies work, cheapest first: index the caption and the paragraph that references it, since most figures are captioned and captions are written to be descriptive; store the OCR text of the image region separately from real text, so a hit is traceable to a guess; generate descriptions at index time with a vision model, which is genuinely effective at a cost of one call per figure; and finally embed the rendered page in a multimodal space, which is the strongest option for visually dense documents and the heaviest to index.

A citation is only worth printing if the chain behind it is unbroken:

document → page 24 → block 7 → chunk 19 → retrieved result → model context → citation

Three rules make that chain hold. One: every chunk carries its block ids and every block carries its page — locators are propagated, never re-derived, because the moment a component recomputes a page number it can be wrong. Two: the locator goes in the prompt, attached to the text, labelled inline; a model cannot cite what it was not told. Three: parse the citations back out of the answer and check each one against the set that was actually retrieved.

Rules one and two make correct citation possible. Rule three is what makes incorrect citation impossible to display — and it is the one people skip.

One extra guard is worth having: a citation whose page contains no retrieved block is a fabrication even when the page exists. Check the block, not just the number.

The second system wins not because it has more components, but because it loses less information and it knows what it does not know.

“Use the LLM for everything” fails on four axes at once: cost, by orders of magnitude; latency, seconds against milliseconds; non-determinism where determinism was available, since parsing the same PDF twice ought to give the same answer; and debuggability. When a parser is wrong you get a stack trace. When a model is wrong you get a plausible paragraph.

The corollary is the direction Alevr has taken: rather than pre-digesting every upload, give the model a sandbox and let it choose the right deterministic tool per question — pypdf here, pandas there, Pillow for a crop — instead of guessing at upload time which one it will eventually need.

What is good today. Dual-channel handling for every provider that supports a native document part, so the PDF itself reaches the model rather than our guess about it. Page rendering in-process, with no system binary to install. A block model with real locators — page, slide, sheet, cell range, line range, heading path, bounding box, confidence. Reading on demand rather than at upload. Tools that let the model navigate a file itself instead of being handed a summary. And a sandbox path that is correctly gated.

What is not there yet, in the order we think it is worth doing:

None of it is a rewrite. Each item is additive: provision a sandbox and set two environment variables; add one call between retrieval and assembly; add a page-asset row; add two columns that extractors fill when they can. The parts that work are not touched.

1. 01Dual channel. Text and pixels, per page, together. The single most important design fact on this page.
2. 02Never gate on text.length > 0. That check is what produces “couldn’t read this file” for documents that read perfectly.
3. 03Read when asked, not when stored. At upload there is no question, no model and no reason. Every verdict made then is a guess that becomes permanent.
4. 04Parsers for parsing, models for understanding. Using a model where a parser works costs a thousand times more and hallucinates.
5. 05A ladder, never a gate. Native parser, better parser, OCR, vision. No single rung’s failure may end a file’s life.
6. 06Locators are the product. Page, section, line. A chunk you cannot locate is a chunk you cannot cite.
7. 07Retrieval is a scaling strategy, not a reading strategy. If it fits, send it whole.
8. 08Hybrid beats vectors. Exact strings need BM25, meaning needs embeddings, and you need both, fused.
9. 09Rerank. The cheapest large quality win in the whole pipeline.
10. 10An explicit locator short-circuits retrieval. “Page 37” is a WHERE clause, not a search query.
11. 11Structure once destroyed is never recovered. No prompt rebuilds a flattened table.
12. 12Distinguish verified text from reconstructed text. OCR carries confidence; embedded text does not need it. Never merge them silently.
13. 13Say what you dropped. Silent truncation manufactures confident wrong answers.
14. 14Validate citations after generation. The only reliable defence against hallucinated page numbers.
15. 15Sandbox anything the model writes, remotely. A prompt injection in a PDF must not become code execution on your host.

Everything tagged Published on this page comes from one of these. Where a vendor has not published something, the page says so rather than filling the gap.

• Anthropic — PDF support
• Anthropic — Files API
• Anthropic — Vision
• Anthropic — Code execution tool
• Google — Gemini document processing
• OpenAI — OpenAPI specification
• Microsoft — Copilot Studio document handling