arXiv

The complete arXiv metadata corpus: every paper's title, abstract, authors, categories, identifiers and dates.

Source and licence

Metadata comes from arXiv's published metadata snapshot, which is CC0 1.0 (public domain). It is mirrored as parquet at librarian-bots/arxiv-metadata-snapshot, which tracks the upstream Kaggle release.

Full text is not included. arXiv's PDFs and LaTeX sources are under the default arXiv licence, which does not grant third parties redistribution rights. Records carry absUrl and pdfUrl that link to arXiv; nothing is copied.

Transform decisions

Four choices shape how this data behaves for anything reading it:

• categories is an array, not a string. arXiv publishes it space-separated ("cs.LG cs.AI stat.ML"). Left that way, every distinct combination becomes its own facet value and filtering is useless. As an array each category is indexed independently.
• authors is an array of display names. Built from authors_parsed ([last, first, suffix]) rather than the raw authors blob, which no consumer can split reliably. Authors stay a field rather than becoming Author records: 3.1M papers would mint millions of them with no reliable identity to merge on, which is a modelling claim this data can't support.
• LaTeX in abstracts is preserved. Roughly a third of abstracts contain $...$ math. It is the real content, and stripping it correctly is lossy guesswork — $O(n\log n)$ has no faithful plain-text form. Consumers that want it rendered or removed can do so knowing the source is intact.
• Control characters are stripped, and internal whitespace collapsed. These are never meaningful and broke a previous ingest.

Category and License are separate record types so they are countable and referenceable rather than repeated opaque strings.

Ingest

npx tsx sources/arxiv/all/ingest.ts fetch    # ~2.9 GB of parquet -> input/
npx tsx sources/arxiv/all/ingest.ts count    # exact row count
npx tsx sources/arxiv/all/ingest.ts push --limit 20000   # shape probe
npx tsx sources/arxiv/all/ingest.ts push     # full corpus

The push streams: it never holds the corpus in memory, uploads the manifest in chunks, and finalizes asynchronously. Both phases are resumable by re-running.

Refreshing

The snapshot is rebuilt weekly upstream. Re-running fetch then push produces a new version; the negotiate protocol transfers only records whose content actually changed. arXiv grows ~32,000 papers/month.