BioFlow

BioFlow is a prototype scientific data-analysis system that explores a specific ML engineering problem: turning a natural-language research question into a reliable, executable analysis workflow over a dataset the system has never seen before. It's an independent, simplified implementation inspired by the natural-language-to-bioinformatics workflow problem that products like Novaflow are built around, it does not reproduce Novaflow's production system, and makes no claim to.

The interesting part of this problem isn't getting an LLM to write Python that plots something. It's making the whole path from question to conclusion trustworthy: knowing what the data actually contains, picking an appropriate method, running it correctly, noticing when a step fails and recovering instead of hallucinating past it, and being able to show your work afterward. BioFlow is a working implementation of that path, plus an evaluation framework that checks whether the path actually holds up rather than just reading the final answer.

---

What it does

1. Upload a scientific dataset (.csv/.tsv or .h5ad).

2. BioFlow structurally infers modality, shape, column/variable types, and which scientific capabilities (regression, clustering, differential expression, ...) are actually available for this data.

3. A natural-language research question is translated into a structured analysis plan by an LLM.

4. The proposed plan is validated - schema, tool existence, capability availability, dependency ordering, and basic scientific plausibility, before anything runs.

5. The validated plan executes through a fixed set of deterministic scientific tools, one step at a time.

6. Every step's artifacts, metrics, figures, and tables are collected with full provenance (tool, parameters, status, duration, outputs).

7. An evidence-grounded report is generated: every substantive claim must cite a real artifact or metric produced during execution.

8. A reproducible Jupyter notebook is reconstructed from the actual execution trace, not generated independently by the model.

9. The whole pipeline can be scored against benchmark cases with known, programmatically verifiable ground truth.

---

Architecture

The central design decision: the LLM never generates or executes arbitrary code.

Natural-language question
        ↓
Dataset understanding (structural, deterministic)
        ↓
LLM workflow planning (structured output, not free text)
        ↓
Multi-stage workflow validation
        ↓
Controlled scientific tool registry
        ↓
Agent execution loop (observe → act → observe, with bounded recovery)
        ↓
Artifacts / metrics / evidence / provenance
        ↓
Evidence-grounded report + reconstructed notebook
        ↓
Evaluation framework

---

Layer by layer, mapped to the actual code:

• Dataset/profile layer (bioflow/core/inspector.py, bioflow/core/dataset.py) — structural inspection: column types, candidate fields, summary statistics.
• Modality/capability detection (bioflow/core/modality.py, bioflow/core/capabilities.py, bioflow/adapters/) — a tabular adapter and a single-cell RNA-seq adapter (AnnData/Scanpy) detect their format structurally and expose what analyses are actually possible for the loaded data. New modalities are added as new adapters, not by branching in the core.
• Scientific tool registry (bioflow/core/tools.py) — every operation the system can perform is a registered, schema-validated Python function. There is no facility for executing dynamically generated code.
• LLM provider abstraction (bioflow/planning/provider.py) — the planner, agent, and reporter depend on an LLMProvider interface, not a vendor SDK. GeminiProvider is the only implementation today, but swapping models or providers doesn't touch the planning, validation, or execution logic.
• Structured workflow planning (bioflow/planning/planner.py) — the model proposes a workflow as a structured object, not prose.
• Multi-stage validation (bioflow/planning/validators/) — schema, tool existence, capability availability, dependency ordering, and scientific plausibility, each an independent, testable stage.
• Agent execution (bioflow/agent/) — an observe → act → observe loop; every proposed action is independently re-validated against real execution state before the registered tool runs.
• Evidence and provenance (bioflow/core/provenance.py, bioflow/reporting/context.py) — a structured record of every tool call, complete enough to reconstruct what happened and why.
• Scientific reporting (bioflow/reporting/) — the model drafts a report; a deterministic validator rejects any claim citing evidence that doesn't exist.
• Notebook reconstruction (bioflow/notebook/) — built from the real execution trace, not generated by the model.
• API/UI (bioflow/api/, frontend/) — a FastAPI backend and a React/TypeScript frontend over the same pipeline the CLI runs.
• Evaluation framework (bioflow/evaluation/) — described below.

---

Key engineering decisions

Constrained execution. The LLM decides what analysis should happen; deterministic scientific tools perform the actual computation. This bounds the blast radius of a bad or malicious model output to "called a registered tool with the wrong arguments" rather than "executed arbitrary code," and makes every step inspectable and unit testable independently of the model.

Structured planning. The model produces a structured workflow, not free-form instructions, so it can be checked against a schema, the tool registry, dataset capabilities, and step dependencies before anything runs, the same discipline validators apply.

Recovery. The agent observes real tool results and can retry or route around a bounded number of failures. Recovery is driven by the model reacting to genuine execution state each turn, not a scripted fallback path.

Evidence-grounded reporting. Reports reference structured evidence generated during execution; a report that cites evidence that was never produced is rejected. Observation and interpretation stay separated.

Reproducibility. The notebook is reconstructed from what actually executed, not independently generated. A step that never ran gets an honest note instead of fabricated code.

Evaluation as a first-class concern. An LLM-generated scientific answer can be fluent while still being wrong, so correctness is checked at the level of the workflow and the evidence underneath the answer, not just whether the final text sounds plausible.

---

Evaluation

bioflow/evaluation/ is a benchmark and scoring framework, not another layer of unit tests. It answers: given a real dataset and a real question, did the system understand the data, choose an appropriate workflow, execute it reliably, recover from failures sensibly, and produce a grounded conclusion?

It scores seven categories per run: dataset understanding, workflow correctness (against a rubric of required/acceptable/forbidden tools and capabilities), execution reliability, recovery behavior, scientific validity (including known-answer checks against planted, documented ground truth — e.g. a synthetic dataset with a real, programmatically verifiable dose-response slope), report groundedness, and notebook reproducibility. Scoring is deterministic wherever possible: it reads real metrics, artifacts, and execution traces rather than asking an LLM to judge the output. A live Gemini call only ever produces the run being evaluated — never scores it.

There are 5 benchmark cases: three tabular (a positive dose-response relationship, a treatment effect with a genuine null feature, and an independent dataset with a negative relationship, included specifically to prove direction-checking isn't hardcoded to expect a positive slope) and two single-cell (a treatment differential-expression case and a cluster/marker case), each with synthetic, seeded, documented ground truth.

BioFlow is also compared against a direct-LLM baseline that gets the same dataset, the same question, and the same controlled tool registry — deliberately given thinner context (no capability gating, no argument schemas, no repair loop) and no ability to observe intermediate results before finishing its plan, but never blocked from calling a real tool. Both systems are scored with the identical evaluator code.

Current state, verified against this repository: 286 tests pass, 7 deselected (opt-in live-Gemini tests), zero failures. The evaluation suite alone is 103 tests passing, 2 deselected. Running python -m bioflow evaluate against the live Gemini API produced scores of 1.0 across most cases in one run, with one single-cell case scoring 0.95 in one run and 1.0 in another. That variance is real and expected — LLM output isn't byte-identical run to run even at low temperature — so a single live run is a sample, not a fixed benchmark number.

---

Example analysis

The prototype has been exercised end-to-end on the public pancreatic endocrinogenesis (day 15) single-cell dataset commonly used in Scanpy tutorials, not Novaflow's internal data, and no claim of parity with it. A real run through the API completes quality control, normalization, PCA, neighborhood graph construction, UMAP, condition comparison, and volcano-plot visualization, and produces a structured report, downloadable tables/figures, and a reproducible notebook rebuilt from that exact execution trace.

---

Running locally

Prerequisites: Python 3.11+, Node 18+, a Gemini API key.

# Backend
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Configure the LLM provider (never commit this file)
echo "GEMINI_API_KEY=your-key-here" > .env

uvicorn bioflow.api.app:app --reload

# Frontend
cd frontend
npm install
cp .env.example .env   # sets VITE_API_BASE_URL=http://localhost:8000
npm run dev

# Tests — fully offline, no API key required
python -m pytest -q

# Opt-in tests against the real Gemini API (requires GEMINI_API_KEY)
python -m pytest -m live_gemini -q

# Evaluation CLI
python -m bioflow evaluate --case tabular_dose_response   # one case
python -m bioflow evaluate                                # all 5 cases
python -m bioflow evaluate --compare --case tabular_dose_response  # vs. baseline

The default test suite (pytest -q) never calls a real LLM — it's excluded via a live_gemini pytest marker configured in pyproject.toml. Anything under evaluate does call the real API, since there's no CLI-level fake-mode flag for the pipeline itself.

You can also drive the core runtime with no LLM at all:

python -m bioflow inspect path/to/dataset.h5ad
python -m bioflow analyze path/to/dataset.h5ad --output runs

Project structure

bioflow/
  core/            dataset abstraction, tool registry, provenance, errors
  adapters/        tabular and single-cell (AnnData/Scanpy) adapters
  planning/        LLM provider abstraction, planner, validators
  agent/           observe/act execution loop, action validation
  results/         assembles execution state into a universal result model
  reporting/       evidence-grounded report generation + validation
  notebook/        reproducible notebook reconstruction
  api/             FastAPI backend
  evaluation/      benchmark cases, evaluators, runner, baseline, CLI
  cli.py           inspect | analyze | plan | execute | evaluate
tests/             mirrors the package layout; unit, integration, and
                   evaluation tests, plus opt-in live-Gemini tests
frontend/          React + TypeScript UI (Vite)

---

Limitations / next steps

• Benchmark coverage is 5 cases across 2 modalities - real, but narrow. A production system would need many more, across more modalities.
• Only two dataset adapters exist (tabular, single-cell RNA-seq). Adding a modality means writing a new adapter, which is a real but unbuilt piece of work per modality.
• Live LLM evaluation scores vary between runs; they are evidence, not a fixed guarantee.
• The API's job store is in-memory and single-process (a small background thread pool, no queue/worker infrastructure); there's no database, no auth, and no multi-tenant isolation.
• There is one LLM provider implementation (Gemini). The abstraction supports swapping providers, but no second provider has been built or tested against it.
• No cloud/HPC deployment story - this runs as a local process.

---

Tech stack

Backend: Python 3.11, FastAPI, Pydantic v2, pandas, NumPy, SciPy, scanpy/anndata/leidenalg (single-cell), matplotlib, nbformat, the Gemini API via google-genai.

Frontend: React 19, TypeScript, Vite, Vitest.

Testing: pytest, pytest-cov, httpx (FastAPI test client).