Quick Start#

This guide builds a complete index bundle from year-wise Parquet files in a few commands.

1. Prepare your input#

Lit2Vec accepts three input kinds:

  • a directory of year-wise Parquet files (corpus_2024_.parquet …),

  • a SQL/DuckDB database,

  • a litsync corpus directory (year-sharded corpus-*.jsonl).

2. Run the full pipeline#

# From year-wise parquet files
lit2vec run -i data/yearly -o data/indices -t parquet

# From a raw litsync corpus (prepare step runs first)
lit2vec run -i litsync_corpus/ -o data/indices --litsync

# Force rebuilding everything and restrict BM25 to selected years
lit2vec run -i data/yearly -o data/indices -t parquet -E -F -B -y 2024,2025

3. Archive and verify#

lit2vec run -i data/yearly -o data/indices -t parquet --zip
lit2vec verify -o data/indices --expect-zip

The proctor checks file integrity, FAISS/PMID alignment, BM25 loadability, and the zip archive (contents + CRC).

4. Or run stages individually#

lit2vec embed -i parquet -d data/yearly -o data/embeddings
lit2vec faiss -d data/embeddings -o data/faiss
lit2vec bm25 -d data/yearly -o data/bm25
lit2vec sql -d data/yearly -o data/metadata

Next steps#