Platforms

Node and servers

Paths, batches and a command line — the conveniences that only make sense where there is a file system.

Everything else in Genom runs in a browser too, and pays for it by refusing to know about paths, file systems and worker threads. @genomdev/genom/node is where that knowledge is allowed, and it is deliberately thin.

server.ts

import { extractFile, extractAll, openFile, hashFile } from '@genomdev/genom/node';

// One file, by path.
const doc = await extractFile('report.docx');
doc.toMarkdown();

// A corpus, with a fixed number in flight. Failures are collected, not thrown:
// ten thousand documents always contain a few nobody can open.
const results = await extractAll(paths, { concurrency: 8 });
for (const { path, document, error } of results) {
  if (error) console.warn(path, error.message);
}

// The hash an address will carry, without parsing the file.
const hash = await hashFile('report.docx');

When the format is known

Reaching for the umbrella means all four parsers are in the module graph. If a route only ever receives Word documents, ask Word directly — nothing else comes with it.

upload.ts

import { extractDocx } from '@genomdev/docx';

export async function POST(request: Request) {
  const doc = await extractDocx(new Uint8Array(await request.arrayBuffer()));
  return Response.json({ markdown: doc.toMarkdown() });
}

The command line

The fastest way to see what extraction does to your own files, and useful in a pipeline on its own.

@genomdev/genom

npx @genomdev/genom md report.docx
npx @genomdev/genom text *.docx --out corpus.txt
npx @genomdev/genom chunks handbook.docx --max-tokens 512 > chunks.jsonl
npx @genomdev/genom json deck.pptx --images reference

Long documents and the event loop

Parsing is synchronous work on a large buffer. On a server handling requests, put it in a worker thread rather than on the main one — extractAll bounds concurrency but does not move work off the thread it is called on.

Cancellation is supported everywhere through signal, and a cancelled parse stops at the next block boundary rather than at the end.

cancel.ts

const controller = new AbortController();
setTimeout(() => controller.abort(), 5_000);

await extractFile('huge.pdf', { signal: controller.signal });