The same file, read two ways at once
On the left, the document as a person sees it. On the right, what a pipeline reads. They are joined by an address, so pointing at either one points at the other — click a word in the extraction, or click the document and watch the extraction follow.
Four outputs, one parse
Not four converters kept in agreement — one intermediate tree with four ways of writing it down. Which is why the Markdown and the blocks never disagree about what the document said.
Blocks
Markdown
Plain text
Chunks
Why the two panes can point at each other
Extraction hands out text; something downstream finds a passage in it and wants to see that passage in the document. Between them there has to be an address that survives another process, another machine and a relayout.
Coordinates cannot do that — a Word document is a flow of paragraphs and has no pages until it is laid out, and where they fall depends on the fonts the machine happens to have. So an address names a node of the model:
apertura:1:9f3ac21b:body/t2/w4/c1/b0/r0+12
└ scheme ┘ │ hash └ flow ┘└──── steps ────┘└ offset
versionThe eight hex digits are a hash of the file’s bytes, and they are why this can be trusted: an address arriving from a different file is detected rather than quietly resolved to whatever node happens to sit at that path. Silent wrongness is how every naive offset scheme fails, and it fails in production rather than in tests.
An anchor is not one address but three, in decreasing precision, following the W3C Web Annotation model. They fail at different times — the first when somebody saves the file, the second when the extraction options change, the third only when the words themselves go — and the resolver reports which one matched, so an application can see its documents drifting before a user reports a highlight in the wrong place.
// Once, when the document is ingested
const doc = await extract(bytes);
for (const chunk of doc.chunks({ maxTokens: 512 })) {
await index.add({ text: chunk.text, meta: chunk.selectors });
}
// Later, when a question is answered
const hit = await index.search(question);
search.show(hit.meta); // scrolled to and highlightedThe things that are easy to get wrong
Each of these is a place where reading the file literally produces something that looks like content and is not.
A chart is a table
A merged cell is filled
A number is what the sheet shows
A running head appears once
Use it
The library is the product; the command line is the front door.
npx @apertura/node md report.docx
npx @apertura/node chunks *.docx --max-tokens 512 > chunks.jsonl