Implementation Details¶

This page describes how the reference implementation turns structured metadata plus a validated codebook into human-readable and stub-format identifiers.

Flowchart¶

Workflow Summary¶

1. Prepare an input table containing normalized metadata fields.
2. Validate the codebook structure against the JSON Schema.
3. Encode metadata into human or stub identifiers with clarid-tools code.
4. Decode identifiers back to structured fields when needed.
5. Optionally generate QR codes from the resulting IDs.

In short, the paper defines the identifier model, while ClarID-Tools implements the operational workflow around that model.

Architecture¶

This architecture describes the reference CLI implementation used to encode, decode, validate, and generate QR codes for ClarID identifiers.

• Language & framework: Perl 5, Moo and MooX::Options. Perl was chosen for efficiency in handling structured text files and for simplicity in a reference CLI implementation.
• Parsing / validation: YAML::XS, Text::CSV_XS, JSON::Validator (codebook validated by JSON Schema).
• QR codes: qrencode (Linux).
• Config: YAML codebook (controlled vocabulary + optional aliases).

The implementation is modular and intentionally straightforward, with a test suite that helps keep behavior reproducible. Equivalent implementations in other programming languages can preserve interoperability by following the same structural rules and YAML-based configuration model.

Design choices (short)¶

• Full externalization of the identifier spec into JSON Schema was tried but became complex (nested regexes and transforms).
• Hybrid approach: core structural rules are implemented in code for clarity; domain vocabularies (species, tissues, assays, aliases) live in the YAML codebook and are schema-validated.
• This keeps parsing deterministic and easier to maintain while retaining configurability.

Normative vs operational details¶

• The identifier semantics are described in the ClarID paper and in the Specification.
• The CLI flags, file formats, and examples in this repository are operational details of the reference implementation.
• The YAML codebook is the main extension point for adapting ClarID-Tools to local vocabularies without changing the identifier model itself.

Encoding / decoding¶

The human and stub formats are fully interoperable but serve different purposes: the human-readable form emphasizes interpretability, whereas the stub form emphasizes compactness for computational workflows.

project / study¶

• Labels like TCGA_AML remain literal unless an alias is declared in the YAML codebook. Add aliases when you need short representations.

subject_id — Base62, fixed width¶

• Numeric subject_id → Base62 (0-9A-Za-z) with fixed width (default: 3) to simplify parsing.
• Options:
• --subject_id_pad_length — decimal padding width for subject_id in human format.
• --subject_id_base62_width — width of the Base62 subject_id field in stub format.
• Example: subject_id = 999 → Base62 G7 → padded to 0G7.
• Capacity: 62^width - 1 unique IDs.

condition (disease)¶

• ICD-10 codes → internal numeric index → Base62 (fixed length, default 3).
• The numeric mapping is derived from the packaged ICD-10 order map distributed with ClarID-Tools (icd10_order.json).
• Human form: multiple conditions separated by +.
• Stub form: condition codes concatenated (no separator); decoding uses reverse mapping.

unzip *zip
# produces: icd10cm-tabular-2026.xml

jq -Rn '
  [ inputs
    | split("\t")
    | select(length == 2)
    | (.[1] as $code
       | ($code | gsub("\\."; ""))  as $key
       | { ($key): .[0] })
  ]
  | add
' /media/mrueda/2TBS/CNAG/Project_ConvertPheno/databases/icd-10/2026/icd10_label_code.tsv > icd10.json

jq '
  to_entries
| sort_by(.key)
| map(.key)
| to_entries
| map({ key: .value, value: (.key + 1) })
| from_entries
' icd10.json > icd10_order.json

species¶

• stub_code declared in the YAML codebook as a codebook-defined species stub.
• Species stub width is defined by the codebook and must be consistent within a given codebook.
• One code (e.g., 00) reserved for unknown.
• Optional tax_code is kept for traceability (not used in stubs).

#!/usr/bin/env python3
ALPHABET = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
BASE = len(ALPHABET)
MAX_VALUE = BASE ** 2

def encode_base62(num: int) -> str:
    if not (0 <= num < MAX_VALUE):
        raise ValueError(f'Number out of range (0 <= num < {MAX_VALUE}), got {num}')
    high = num // BASE
    low = num % BASE
    return ALPHABET[high] + ALPHABET[low]

tissue, sample_type, assay¶

• Use predefined stub_codes from the codebook (recommended 2–5 chars).
• Decoding strategy: parse fixed-width fields first, then greedy reverse lookup on remaining stub codes sorted by descending length to avoid prefix collisions (e.g., PB, T, HI parse PBTHI correctly).

Extensibility & pragmatic workarounds¶

• Targets subject and biosample entities; extensible to cohorts, datasets, experiments with minor code changes.
• Temporary workaround: repurpose unused codebook fields (e.g., use tissue for geographic location). This keeps identifiers functional if overall structure is preserved.

Implementation notes & tips¶

• Keep stub codes short (2–5 chars) and unique to avoid parsing ambiguity. ✅
• Increase --subject_id_base62_width before cohort size exceeds 62^width - 1.
• Use YAML codebook aliases for stable short labels.
• Maintain the JSON Schema when editing the codebook.
• Re-run encode/decode examples after changing the codebook so documentation examples stay in sync with the current release.