Lit2Vec#

Embedding and search-index toolkit for biomedical literature

Lit2Vec converts biomedical literature corpora (primarily PubMed/MEDLINE, typically produced by litsync) into a ready-to-serve retrieval bundle: dense embeddings in year-wise HDF5 files, quantized FAISS indices, a monolithic Annoy index, BM25 keyword indices, and per-year metadata databases.

Quick Start

Install Lit2Vec and build your first index bundle in minutes.

Quick Start
User Guide

Pipeline stages, output layout, resume behavior, and best practices.

User Guide
CLI Reference

Complete reference for lit2vec commands and options.

CLI Reference
API Reference

Python API for embeddings, indices, metadata, and verification.

API Reference

What Lit2Vec does#

  1. Prepare — normalizes a litsync JSONL corpus into year-wise Parquet files (streaming, bounded memory).

  2. Embed — encodes titles/abstracts with a SentenceTransformer model (default NovaSearch/stella_en_400M_v5) into year-wise HDF5 files.

  3. FAISS — builds per-year 4-bit scalar-quantized ANN indices.

  4. Annoy — builds one monolithic Annoy index over all years.

  5. BM25 — builds sparse lexical indices for title+abstract and authors.

  6. Metadata — exports per-year SQLite databases and PMID Parquet lists aligned with the FAISS rows.

  7. Archive & verify — optionally zips the bundle and validates it with the built-in proctor.

lit2vec run -i data/yearly -o data/indices -t parquet --zip
lit2vec verify -o data/indices --expect-zip

Interrupted runs resume where they stopped: completed years are skipped, partial years are continued, and stale per-year indices are rebuilt — see User Guide.

Where to go next#