CLI Reference

LitSync ships two commands: litsync (the mirror) and litsync-extract (corpus extraction).

litsync

litsync --data-root PATH --email EMAIL [options]

Options

Flag	Description
--data-root PATH	Required. Root directory for the local mirror.
--email EMAIL	Contact email sent in the User-Agent. Defaults to $NCBI_EMAIL; one of the two is required.
--sources ...	Corpora to mirror: pubmed pmc fda clinicaltrials (default: pubmed pmc).
--pmc-groups ...	PMC OA groups: oa_comm oa_noncomm oa_other (default: all three).
--pmc-formats ...	PMC formats: xml txt (default: xml).
--fda-endpoints ...	openFDA endpoints to mirror, e.g. drug/event drug/label (default: all endpoints).
--workers N	Concurrent downloads (default: 4). Keep modest; be polite.
--max-retries N	Per-file retry attempts (default: 5).
--timeout SECONDS	HTTP timeout (default: 60).
--dry-run	Plan only; download nothing.
--reverify	Re-download already-downloaded files to verify integrity.
--prune	Delete local files no longer present on the server.
--count-articles	Count articles in already-downloaded files and exit (no network).
--no-rich	Disable Rich progress bars; use plain text output.
--verbose, -v	Enable debug logging.

Exit code is non-zero if any file failed, so cron/monitoring can alert.

Examples

# Full mirror of all four sources
litsync --data-root /data/literature --email you@institute.org \
  --sources pubmed pmc fda clinicaltrials

# Only two openFDA endpoints
litsync --data-root /data/literature --email you@institute.org \
  --sources fda --fda-endpoints drug/event drug/label

# Daily cron entry (02:30)
30 2 * * *  /path/to/venv/bin/litsync --data-root /data/literature \
  --email you@institute.org >> /data/literature/_state/cron.log 2>&1

litsync-extract

litsync-extract --data-root PATH --out PATH [options]

Options

Flag	Description
--data-root PATH	Mirror root to read (default: ./data/literature).
--out PATH	Output directory for sharded JSONL (default: ./data/corpus).
--sources ...	Sources to extract (default: pubmed pmc).
--shard-size-mb N	Maximum shard size in MB (default: 256).
--limit N	Extract at most N files per source (quick tests).
--resume / --no-resume	Skip source files already recorded in resume state (default: on).
--reset	Clear resume state and re-extract from the beginning.
--yearly	Split output into subdirectories by record year.
--require-abstract	Skip records with an empty abstract.
--require-body	Skip records with an empty body.
--require-text	Skip records with neither abstract nor body text.
--no-rich	Disable Rich progress bars.
--verbose, -v	Enable debug logging.

Examples

# Full corpus extraction (hours; very large)
litsync-extract --data-root /data/literature --out /data/corpus \
  --sources pubmed pmc fda clinicaltrials

# Quick check: one file per source
litsync-extract --data-root /data/literature --out ./data/corpus_test \
  --sources pubmed pmc fda clinicaltrials --limit 1

Makefile shortcuts

The repository Makefile wraps both commands:

Target	Description
make install / make dev	Install requirements / editable package.
make sync EMAIL=you@x.org	Run the incremental mirror.
make dry-run	Plan only, download nothing.
make reverify / make prune / make count-articles	Maintenance operations.
make pubmed / make pmc / make fda / make clinicaltrials	Sync a single source.
make extract / make extract-test	Full extraction / one-file-per-source test.

Environment variables

NCBI_EMAIL

Default contact email when --email is not given.