LitSync#
Incremental mirror for PubMed, PMC, FDA, and ClinicalTrials.gov
LitSync is a daily-runnable CLI for mirroring bulk biomedical datasets. It tracks every file in a SQLite state database, so re-runs do the minimum work: already-verified immutable files are skipped with no network request beyond the directory/manifest listing.
Install LitSync and run your first mirror in five minutes.
Learn about the on-disk layout, integrity model, cron setup, and best practices.
Complete reference for litsync and litsync-extract commands and options.
Python API for configuration, syncing, sources, state, and extraction.
What LitSync does#
Plan — lists the remote directories/manifests of the enabled sources.
Diff — compares each remote file against the SQLite ledger and local disk.
Download — fetches only new or changed files, atomically and resumably.
Verify — checks MD5 sidecars (PubMed) or size/ETag (PMC, FDA, ClinicalTrials.gov).
Extract — converts the mirror into sharded JSONL corpora with
litsync-extract.
litsync --data-root /data/literature --email you@institute.org
Where to go next#
Installation — set up the package.
Quick Start — run a first mirror and extract a test corpus.
User Guide — understand the mirror layout and integrity model.
CLI Reference — look up every command and flag.
API Reference — use LitSync as a Python library.