Leakage-safe multi-omics audit and fusion benchmarking

What omicau answers

Before anyone builds a model on a multi-omic dataset, omicau answers three questions about the data itself:

1. Does combining omic layers actually help, or does one layer already carry the signal?
2. Is the apparent signal trustworthy, or an artifact of batch effects, outcome-linked missing data, or information leakage in the cross-validation?
3. What should you do next — adopt the fusion model, prefer a single layer, distrust a confounded layer, or re-check the splits?

It makes no patient predictions and issues no diagnosis. Under leakage-safe, group-aware cross-validation it benchmarks classical and neural fusion with group-level bootstrap 95% confidence intervals, tests each layer's marginal contribution with a corrected resampled t-test, measures cross-layer redundancy, benchmarks negative controls to prove the pipeline does not leak, and locks provenance with a value-level SHA-256 hash.

Installation

omicau is on PyPI — one line installs it. Python ≥ 3.10; every option below gives the same omicau command.

pip / pipx simplest

pipx install omicau            # isolated CLI install — recommended for a tool
pip install omicau             # or into your current environment
pip install "omicau[all]"      # + every cross-platform runtime feature

conda / mamba

git clone https://github.com/tunabirgun/omicau.git && cd omicau
conda env create -f environment.yml     # or: mamba env create -f environment.yml
conda activate omicau

From source development

git clone https://github.com/tunabirgun/omicau.git && cd omicau
pip install ".[all,dev]"

Optional extras — combine freely: omicau[x] from PyPI, .[x] from a checkout, or (on a pipx install) pipx inject omicau <deps>: [llm] (AI verdict: Claude / ChatGPT / Gemini / local), [data] (remote data hubs), [ui] (no-code web UI), [cptac] (CPTAC hub; needs a build toolchain), [all] (every cross-platform feature), [dev] (tests). omicau's error messages print the exact install command for your environment.

Core dependencies: numpy, pandas, scipy, scikit-learn, torch, plotly, click, jinja2, tqdm. Releases are automatic: bumping the version in pyproject.toml and merging to main publishes to PyPI via OIDC (no stored token), and conda-forge auto-follows PyPI.

Quick start

omicau check-env                                 # CPU/GPU + dependency status
omicau bootstrap --dataset mock --out-dir demo   # write a synthetic dataset + config
omicau run --config demo/config.json --cores 8   # full audit -> demo/run/report.html
omicau verify --config demo/config.json          # recompute the provenance hash

Open demo/run/report.html for the dashboard. demo/run/audit.json is the full machine-readable state; model_metrics.csv, modality_ledger.csv, and missingness_tests.csv are flat exports; MODEL_CARD.md is a research-use-only model card.

The mock dataset is offline and the recommended starting template — it writes a runnable config.json and example CSVs with deliberately redundant, batch-confounded, and pure-noise layers, so the diagnostics have something to flag. Add --task regression for a regression mock.

Bring your own data

Provide one file per modality, one clinical table, and a small config that names them (paths resolve relative to the config). Ingestion is forgiving: delimiter and orientation are auto-detected (samples × features or features × samples), sample names are fuzzily matched across files, and common NA tokens, European decimals, and ±inf are healed. Missing values stay masked — never imputed at ingest.

The clinical table is one row per sample: a sample-id column, the target, and optional group (e.g. patient id, for leakage-safe splitting) and batch columns. A single modality is allowed — cross-modality comparisons are simply skipped. The feature-id examples above are human, but any organism's identifiers work unchanged — see Model & non-model organisms.

{
  "run_name": "my_study", "output_dir": "run",
  "modalities": [
    {"name": "rna",     "path": "rna.csv"},
    {"name": "protein", "path": "protein.csv"}
  ],
  "clinical": {"path": "clinical.csv", "target": "outcome",
               "sample_id": "sample_id", "group": "patient_id",
               "batch": "batch", "task": "auto"},
  "cv": {"n_splits": 5, "seed": 42, "n_bootstrap": 1000}
}

Model & non-model organisms

Feature IDs are yours to name

Feature identifiers are treated as opaque strings — Zm00001d027230, FBgn0000490, AT1G01010, an OTU id, a peak m/z, or a de-novo transcript name all work identically to a human gene symbol. Nothing is mapped, matched to a reference, or validated against a species. Only sample names are matched across your files (to line the tables up); feature names are carried through verbatim.

Sample-name normalization is off by default

"normalization": {"preset": "tcga"}

Grouping and batch: the pseudoreplication guard

The most important columns for animal and plant work are group and batch in the clinical table. They stop the same thing that patient-id grouping stops in a clinical cohort: samples that aren't independent leaking optimistic scores.

• group — the unit that must never be split across training and testing. Set it to whatever your replicates share: the animal id when you have several tissues or timepoints per animal, or the litter / cage / plant / plot when that is the unit you actually randomized. omicau keeps every sample from one group entirely in train or entirely in test.
• batch — the nuisance structure to watch for: sequencing run, extraction day, plate, growth chamber. omicau flags a data type only when its variance tracks the batch and the batch tracks the outcome.

Cage can be either: use it as group if you treated whole cages, as batch if cages are just where animals were housed.

A minimal mouse config differs from the human one only in the column names it points at:

{
  "run_name": "mouse_liver", "output_dir": "run",
  "organism": "Mus musculus",
  "modalities": [
    {"name": "rna",     "path": "rna.csv"},
    {"name": "protein", "path": "protein.csv"}
  ],
  "clinical": {"path": "clinical.csv", "target": "genotype",
               "sample_id": "sample", "group": "animal_id",
               "batch": "seq_run", "task": "classification"},
  "cv": {"n_splits": 5, "seed": 42, "n_bootstrap": 1000}
}

To try a real non-human dataset in one command, use the cross-organism data hub: omicau bootstrap --dataset expression_atlas --out-dir d (defaults to a mouse experiment from EMBL-EBI Expression Atlas).

The optional no-code UI

omicau is a CLI tool first. For non-coders there is an opt-in local web UI — a wizard to upload files, map columns, and run the audit without a config file:

pip install ".[ui]"
omicau ui        # opens a browser wizard on 127.0.0.1

It binds 127.0.0.1 behind a one-time token, opens a browser, and walks through: drop files → confirm the auto-guessed omic role and orientation → map the clinical columns (each dropdown shows its live consequence, e.g. class balance or the leakage implication of the grouping) → check cross-file alignment → run the identical pipeline → read the report in-page.

AI plain-language verdict optional

omicau always writes a deterministic, rule-based summary of the audit. On top of that you can ask an AI model to turn the numbers into a short plain-English verdict — naming the best fusion model, whether combining layers helped, which layers to trust, and what to do next. This is optional and additive: skip it and nothing changes; the built-in summary and every figure are still produced.

Pick a provider

The same audit; you choose who writes the prose. Cloud models need an API key you paste once; a local model needs nothing but the server address.

The model id is a free-text field prefilled with a sensible default — paste any model your account or local server supports.

In the no-code UI

On the Options step, tick “Add an AI plain-English verdict”, pick a service, and either paste your key (cloud) or confirm the server address (local). Run the audit as usual; the verdict appears in the report next to the rule-based summary.

Easiest path: a local model, no key, fully offline

1. Install Ollama (Windows / macOS / Linux) and open it.
2. In a terminal, download a model once: ollama pull llama3.1
3. In omicau's Options step, choose “Local model on this computer”. The server address http://localhost:11434/v1 is prefilled. Leave the model as llama3.1 (or whatever you pulled).
4. Run the audit. The verdict is generated locally.

From the CLI / config

The UI and CLI produce identical results. In config.json, add an llm block; the key is read from a named environment variable (never stored in the file):

"llm": {
  "enabled": true,
  "provider": "anthropic",              // anthropic | openai | gemini | local
  "model": "claude-sonnet-5",
  "api_key_env": "ANTHROPIC_API_KEY",   // env-var NAME — never the key itself
  "base_url": null                       // set for local, e.g. http://localhost:11434/v1
}

export ANTHROPIC_API_KEY=sk-...      # cloud providers read the named env var
omicau run --config config.json      # local models need no key
pip install ".[llm]"                  # installs the anthropic + openai SDKs

Reading the report

The dashboard has two tabs, both driven by the same audit.json. Executive summary answers the three questions in plain language and states a verdict and a trust checklist — read this first. Research detail holds the numbers behind those answers.

The answer strip

Three questions, one answer each: Does combining layers help? · Is the data trustworthy? · What should you do next? Under it, the clinical verdict is one plain paragraph naming the best fusion model, its headline score, whether fusion beat the best single layer, which layers carry independent signal, and which to treat with caution.

The run report card

A short pass / caution / fail checklist telling you how far to trust the run itself, before you trust any score in it:

• ✓ pass Control baselines at chance — a shuffled target is not predictable, so no target/pipeline leakage.
• ✓ pass Samples keyed to independent subjects — a group id keeps each subject's samples wholly in train or test, so repeated samples can't leak; a caution here means no group column was set.
• △ caution Batch confounding / missingness — a layer whose batch is confounded with the outcome, or missingness linked to the outcome.
• ▲ fail Leakage flagged — a control beat chance; investigate the splits before trusting any score.

Is my result trustworthy?

Read the result only as far as these allow:

How it works

The whole pipeline is designed to not fool you. Preprocessing (median-impute → variance-filter → standardize → feature selection) is fitted inside each training fold only; cross-validation is group-aware (multiple samples from one patient never split across train and test).

Fusion across the integration axis

omicau benchmarks three integration regimes so you can see which a dataset wants:

• Early — feature concatenation, per estimator (logistic/ridge, random forest, gradient boosting).
• Intermediate — a masked global-pooling neural network that ignores missing features per sample (no artificial variance injected).
• Late — stacking: a meta-learner cross-validated over the single-modality out-of-fold predictions (leakage-safe by construction).

Each layer's marginal gain — how much it adds beyond the others — is tested with the Nadeau–Bengio corrected resampled t-test (k-fold train sets overlap, so a naive paired t-test has inflated Type-I error). A layer is only badged predictive (adds signal) when that gain clears both an effect-size floor and the significance test — a positive but non-significant gain reads "not additive", so a noisy gain is never shown as an established contribution. Cross-layer redundancy is the linear CKA; a layer highly aligned with a stronger one is flagged redundant.

Validation

Each score matches the literature — PAM50 subtypes are transcriptionally defined (Parker 2009; TCGA 2012), expression predicts SOX10 dependency in melanoma lineage (DepMap), the Dlx3 knockout is a moderate signature so 0.778 with a wide CI at n=12 is the honest read (Duverger 2014), and the yeast osmostress and Arabidopsis defense programs are large real responses (Babazadeh 2017; Liu 2015). Where n ≪ features (zebrafish, Arabidopsis), a perfect AUROC is a small-sample artifact: the shuffled-target control also beats chance, so omicau flags leakage and refuses to certify the score rather than reporting a misleading 1.000. That refusal — not the headline number — is the point of the tool.

Uncertainty & calibration

• Confidence intervals — every model's primary metric carries a group-level bootstrap 95% CI (resampling whole patient groups). The per-fold spread is shown only as "fold dispersion," since correlated folds bias it low — often several-fold narrower than the true CI.
• Calibration (binary classification) — a reliability curve, Brier score, and ECE, flagging that class_weight="balanced" miscalibrates probabilities by construction.
• Feature attribution — leakage-safe permutation importance on held-out folds, reported as mean ±SD across folds. It is unconditional and biased under feature collinearity (Hooker 2021; Strobl 2008), so the ranking is indicative, not causal.

Data hygiene & generalization

• Missingness bias — Kruskal-Wallis / Spearman / chi-square tests (BH-corrected) for missingness associated with the outcome (possible MNAR).
• Batch & confounding — PCA silhouette + ANOVA η² for batch structure, plus a chi-square/Cramér's V (categorical) or ANOVA/η² (continuous) test for batch–outcome confounding. A layer is called batch-confounded only when its variance is batch-structured and batch is confounded with the outcome (an orthogonal batch effect is harmless).
• Control baselines — shuffled-target / shuffled-feature / random-noise runs; the leakage alarm fires when a control's CI lower bound clears chance. These catch target/pipeline leakage; group leakage from a mis-set group column is surfaced separately as a grouping-status check.
• Per-site / subgroup performance — the best model re-scored within each site/subgroup, with the max-minus-min gap and each stratum reported with a bootstrap 95% CI (strata below n=20 shown but not scored), so a disparity is only called out when its interval excludes a negligible difference.
• Cross-site stress test (opt-in) — folds blocked on batch (leave-one-batch-out) for an honest new-batch estimate + optimism gap.

Reproducibility & provenance

omicau verify --config config.json --expected <hash>   # exit 1 on mismatch
omicau verify --config config.json --audit run/audit.json

Data hubs

Optional clients pull public cohorts (verified live). The primary workflow remains your own data.

omicau bootstrap --dataset xena --out-dir d --preset brca --target PAM50Call_RNAseq

CLI & configuration

Key config knobs: cv.n_splits, cv.n_bootstrap, cv.batch_blocked, compute.deterministic, neural.enabled, llm.enabled. Configs may be JSON or TOML; YAML additionally needs pip install omicau[yaml].

Research use only

omicau · github.com/tunabirgun/omicau · part of the Build with Claude: Life Sciences hackathon (Anthropic × Gladstone Institutes).