CLI

bff-tools {validate,vcf,tsv,demo,doctor,install-resources,test,compare} [options]

Run bff-tools <command> --help for the installed version. Conversion creates a new project directory and never overwrites an existing one.

validate

With XLSX input, this command converts each populated worksheet into a BFF JSON collection, validates its records, and writes valid collections to the output directory. With JSON input, it validates the existing collections without rewriting them.

Export the packaged template:

bff-tools validate --template-out metadata.xlsx

Build and validate BFF JSON from a workbook:

bff-tools validate -i metadata.xlsx -o bff

Validate one or more collection files:

bff-tools validate -i individuals.json biosamples.json

Option	Meaning
-i, --input FILE ...	One XLSX workbook or one or more named JSON collections
--template-out PATH	Export a fresh metadata workbook instead of validating
-o, --out-dir DIR	XLSX serialization destination; created when absent
-s, --schema-dir DIR	Override the packaged dereferenced schemas
--gv	Include the workbook genomicVariations sheet or JSON collection
--gv-vcf	Stream generated genomicVariationsVcf.json[.gz] or .jsonl[.gz] records
--check-schema	Self-validate all schemas when used alone, or each input-selected schema before checking data
--ignore-validation	Write workbook output despite validation issues
--verbose	Print progress for large inputs

Validation exits nonzero when schema issues are found, unless they are explicitly ignored. bff-tools validate --check-schema checks the complete schema registry without requiring input data. Combined with --input, it checks only the selected schemas before ordinary record validation.

demo

Exercise the installed converter, schema validator, and standalone browser using a packaged, fully annotated fixture:

bff-tools demo

The default destination is a new bff-tools-demo/ directory. Use --output-dir DIR to select another path or --no-browser to generate only BFF. The command requires no external annotation resources because it does not rerun annotation; it is an onboarding example, not a raw-VCF integration test.

doctor

Check packaged assets and the annotation profile selected for the current environment without executing a pipeline:

bff-tools doctor --genome hg38

The default profile is hg19; b37 is accepted as an alias for hs37. Use -c FILE or --config FILE to inspect a custom layout and --no-color for plain output. Configuration and data-root precedence match conversion commands.

An installation without external resources exits successfully as CORE READY (annotation not configured). Missing or invalid paths selected explicitly through BFF_TOOLS_DATA, BFF_TOOLS_CONFIG, or --config exit nonzero. READY means the selected raw-annotation profile also passed.

install-resources

Install the maintained external annotation bundle into the directory selected by BFF_TOOLS_DATA:

export BFF_TOOLS_DATA=/absolute/path/to/beacon2-cbi-tools-data
bff-tools install-resources

Pass --data-dir DIR instead of exporting the environment variable, or use --print-links to list the public Google Drive files for manual download. The command reuses existing files, verifies every archive part, assembles and extracts the bundle, and creates its writable tmp/ directory.

test (development)

Developers and bundle maintainers can run the packaged compact annotation integration test against the selected external bundle:

export BFF_TOOLS_DATA=/absolute/path/to/beacon2-cbi-tools-data
bff-tools test

The command annotates the packaged chromosome 1 1000 Genomes fixture, validates the resulting BFF, and compares all records semantically with the versioned reference output. It does not run the external CINECA chromosome 22 fixture. It is a development and bundle check, not a required user workflow. Use --data-dir DIR instead of the environment variable, --threads N for annotation, or --output-dir DIR to retain the generated project. Add --verbose for detailed pipeline output or --no-color for plain stage labels.

The release-scale chromosome 22 procedure uses the regular bff-tools vcf and bff-tools validate commands plus the installed bff-tools compare command. Do not use plain diff or compare compressed-file checksums for this parity gate. See Full CINECA Release Fixture.

compare

Compare two BFF genomic-variation files semantically. The command streams compressed or uncompressed JSON, ignores run-specific provenance and known order-only differences, and exits nonzero with the first differing record and JSON path:

bff-tools compare \
  --expected reference/genomicVariationsVcf.json.gz \
  --actual run/vcf/genomicVariationsVcf.json.gz

vcf

Annotate and convert raw VCF input:

export BFF_TOOLS_DATA=/absolute/path/to/beacon2-cbi-tools-data
bff-tools vcf -i cohort.vcf.gz --genome hg38 --dataset-id cohort-1

Annotation is enabled by default because the converter requires a compatible SnpEff ANN header. Pass --no-annotate only when the input VCF is already annotated and already normalized to one ALT allele per record (biallelic records). This option bypasses the bcftools normalization stage as well as annotation. Raw input requires the external annotation bundle selected through BFF_TOOLS_DATA.

bff-tools vcf -i cohort.annotated.vcf.gz \
  --genome hg38 --dataset-id cohort-1 --no-annotate

The input may be plain .vcf or gzip/BGZF-compressed .vcf.gz. Single-sample and multi-sample VCFs are supported. gVCFs must first be genotyped or converted to a standard variant VCF.

tsv

bff-tools tsv -i genotypes.txt.gz --sample-id sample-1 --genome hg19

TSV conversion creates a VCF intermediate, annotates it, and then uses the same VCF-to-BFF converter. Annotation cannot be disabled for TSV input.

Conversion Options

Option	Meaning
-i, --input FILE	Input VCF, TSV, or supported compressed equivalent
-p, --param FILE	Optional YAML parameters
-c, --config FILE	Override the packaged external-tool and annotation-resource layout
-o, --project-dir DIR	Explicit new run directory
-t, --threads N	Positive thread count passed to external stages and compression
--genome NAME	hg19, hg38, hs37, or b37
--dataset-id ID	Dataset identifier embedded in BFF records
--sample-id ID	Sample identifier used by TSV conversion
--annotate, --no-annotate	Annotation is enabled by default; disable only for a compatibly annotated, normalized biallelic VCF
--browser, --no-browser	Enable or disable standalone HTML generation
--jsonl, --no-jsonl	Write JSON Lines (.jsonl.gz) instead of the default JSON array
--verbose	Stream stage output rather than showing the interactive spinner
--progress-every N	With --verbose, report VCF progress every N records (default: 10,000)

Values supplied directly on the command line override parameter YAML values. YAML values override built-in defaults.

BFF_TOOLS_DATA overrides the {base} root in the selected resource layout. Layout selection is explicit --config, then BFF_TOOLS_CONFIG, then the repository or packaged default. Absolute paths in a custom layout remain unchanged.

The Python VCF-to-BFF conversion itself is single-process and streaming. Increasing -t helps only stages that support threads; it does not partition records across Python workers.

Standard BFF JSON arrays remain the default for compatibility. Use --jsonl when a downstream tool such as mongoimport benefits from one complete JSON document per line. Browser generation and validate --gv-vcf accept either generated format.

For finer-grained diagnostics on a short file, combine --verbose with a smaller interval, for example --progress-every 100. The same option is available when running src/bff_tools/vcf2bff.py directly. Progress is also retained in <project>/vcf/run_vcf2bff.log.

Common Options

Option	Meaning
--verbose	Print stage output and converter progress instead of the interactive spinner
--debug N	Preserve detailed execution output for diagnosis
-nc, --no-color	Disable ANSI colors
-ne, --no-emoji	Disable emoji output
-V, --version	Print the application version

Exit Behavior

• argument, configuration, preflight, and pipeline failures exit nonzero;
• validation issues exit nonzero unless --ignore-validation was explicitly supplied;
• a VCF without a usable SnpEff ANN header exits nonzero with instructions to annotate it;
• stage failures name the generated log file to inspect.

Removed Commands

The former load and full commands were retired in 2.0.13. Data preparation remains in bff-tools; database deployment remains independent. See MongoDB Import for the preserved indexing and loading procedure.