Skip to content
Apertura
Extraction lab

The same file, read two ways at once

On the left, the document as a person sees it. On the right, what a pipeline reads. They are joined by an address, so pointing at either one points at the other — click a word in the extraction, or click the document and watch the extraction follow.

Loading the lab…
Nothing is uploaded. The samples are fetched once as static bytes; a file of your own is read in this tab and never leaves it. Open the network panel first if you would rather see that than read it.

Four outputs, one parse

Not four converters kept in agreement — one intermediate tree with four ways of writing it down. Which is why the Markdown and the blocks never disagree about what the document said.

Blocks

A typed tree: headings, lists, tables with their rows, images, maths — and charts as the numbers they plot rather than a hole where a picture was. What a program gets, as opposed to a string.

Markdown

GFM, written to be read. Tables keep their alignment, a pipe inside a cell does not end the column, a paragraph starting “1972.” does not silently become a list, and the document’s own multi-level numbering survives as `2.3.1` rather than collapsing to `1.`

Plain text

Genuinely plain: no markers, no markup, nothing that would become a false hit in a search index. The only characters that are not from the document are the separators between blocks.

Chunks

Cut on the structure rather than every 512 characters: a table row is never split, a large table repeats its header in each piece, and the heading path travels with the text so it retrieves.
Addressing

Why the two panes can point at each other

Extraction hands out text; something downstream finds a passage in it and wants to see that passage in the document. Between them there has to be an address that survives another process, another machine and a relayout.

Coordinates cannot do that — a Word document is a flow of paragraphs and has no pages until it is laid out, and where they fall depends on the fonts the machine happens to have. So an address names a node of the model:

A locator
apertura:1:9f3ac21b:body/t2/w4/c1/b0/r0+12
└ scheme ┘ │ hash  └ flow ┘└──── steps ────┘└ offset
           version

The eight hex digits are a hash of the file’s bytes, and they are why this can be trusted: an address arriving from a different file is detected rather than quietly resolved to whatever node happens to sit at that path. Silent wrongness is how every naive offset scheme fails, and it fails in production rather than in tests.

An anchor is not one address but three, in decreasing precision, following the W3C Web Annotation model. They fail at different times — the first when somebody saves the file, the second when the extraction options change, the third only when the words themselves go — and the resolver reports which one matched, so an application can see its documents drifting before a user reports a highlight in the wrong place.

rag.ts
// Once, when the document is ingested
const doc = await extract(bytes);
for (const chunk of doc.chunks({ maxTokens: 512 })) {
  await index.add({ text: chunk.text, meta: chunk.selectors });
}

// Later, when a question is answered
const hit = await index.search(question);
search.show(hit.meta);   // scrolled to and highlighted
49 374
chunks measured
across 2 268 real documents of the test corpus
100%
resolved by exact address
every chunk found its way back, and the text matched
0
dependencies
no Python, no WASM, no native binary

The things that are easy to get wrong

Each of these is a place where reading the file literally produces something that looks like content and is not.

A chart is a table

The numbers behind a quarterly report’s headline are stored in the file, because that is how a chart survives being opened away from its workbook. Every other extractor in this space treats a chart as a picture, and a picture as nothing.

A merged cell is filled

Excel keeps the value in the top-left of a merge and nothing anywhere else. Read naively, a header row comes out as one word and nine empty columns — and no reader, human or otherwise, can tell which column a figure belongs to.

A number is what the sheet shows

The cell holds 45306 and the sheet says 15 January 2024. A search for the date has to find it, which means running every value through its number format rather than reading storage.

A running head appears once

A header repeated on eighty pages contributes eighty identical fragments, and identical fragments are the worst thing that can happen to a nearest-neighbour search — they crowd out the answer with copies of the company name.

Use it

The library is the product; the command line is the front door.

npx @apertura/node md report.docx
npx @apertura/node chunks *.docx --max-tokens 512 > chunks.jsonl
Read the docsBuilding retrievalThe viewer on its own