Auto-generate dependency graphs and workflow visualizations

[jd-garv]

Overview

Automatically generate visual dependency graphs from BioSpec specifications to show analysis workflow structure, data flow, and relationships between project entities.

Motivation

Current state:

• Dependencies listed as text in intent.md (e.g., "InferCNV cannot be run without cell annotations")
• Relationships described in relationships.md (RQ → Dataset mappings)
• No visual representation of workflow structure or bottlenecks
• Difficult to identify parallel vs sequential analyses
• Hard to communicate project structure to stakeholders

Desired outputs:

• Visual dependency graphs showing analysis workflow
• Data flow diagrams showing how datasets feed into analyses
• RQ-Dataset-Analysis relationship maps
• Interactive or static visualizations embedded in docs

Proposed Visualization Types

1. Analysis Pipeline DAG (Directed Acyclic Graph)

Shows: Sequential and parallel analysis steps

Example visualization:

Raw Data → QC → Normalization → Batch Correction
                                       ↓
                                   Clustering ←→ Marker ID
                                       ↓              ↓
                                   UMAP/tSNE    Cell Type Anno
                                       ↓              ↓
                                       └──→ DE Analysis
                                                ↓
                                           Pathway Analysis
                                                ↓
                                           Visualization

Generated from:

• intent.md: Analysis Objectives with dependencies
• intent.md: Milestones (high-level phases)
• project_overview.md: Computational Activities in Scope

2. Dataset-RQ-Analysis Network

Shows: Which datasets feed which research questions via which analyses

Example visualization:

[Dataset 1: scRNA-seq] ──┐
                         ├─→ [RQ1: TME Composition] ─→ [Clustering + DE]
[Dataset 2: Visium]  ────┤                              
                         └─→ [RQ2: Spatial Patterns] ─→ [Spatial Stats]

[Dataset 3: Public Ref] ─────→ [RQ1: TME Composition] ─→ [Reference Mapping]

Generated from:

• relationships.md: RQ vs Datasets mapping
• intent.md: Research Questions
• datasets.md: Dataset list

3. Data Flow Diagram

Shows: How data transforms through pipeline stages

Example visualization:

┌─────────────┐
│ FASTQ Files │
└──────┬──────┘
       ↓ [alignment]
┌─────────────┐
│  BAM Files  │
└──────┬──────┘
       ↓ [quantification]
┌─────────────┐
│Count Matrix │
└──────┬──────┘
       ↓ [QC filtering]
┌─────────────┐
│Filtered Data│
└──────┬──────┘
       ↓ [normalization]
┌─────────────┐
│Seurat Object│
└──────┬──────┘
       ├─→ [clustering] → Clusters
       ├─→ [dim reduction] → UMAP
       └─→ [DE] → Gene Lists

Generated from:

• datasets.md: Data types and processing levels
• intent.md: Analysis objectives (processing steps)
• Inferred intermediate data types

4. Resource-Analysis Dependency

Shows: Which analyses require which computational resources

Example visualization:

                         ┌─ HPC (16 cores, 64GB)
                         │
QC → Normalization → Clustering → DE → Pathway
↑         ↑             ↑         ↑       ↑
│         │             │         └─ HPC (8 cores, 32GB)
│         │             └─ HPC (16 cores, 128GB) **bottleneck**
│         └─ Local (4 cores, 16GB)
└─ Local (4 cores, 8GB)

Generated from:

• project_resources.md: Hardware specs
• intent.md: Analysis objectives
• Estimated resource needs per analysis type

Markdown-Based Visualization Approaches

Approach 1: Mermaid.js (Recommended)

Pros:

• Native GitHub rendering
• Supports flowcharts, sequence diagrams, Gantt charts
• Declarative syntax
• Widely supported (GitHub, GitLab, VS Code)

Example Mermaid syntax for DAG:

graph TD
    A[Raw FASTQ] --> B[Quality Control]
    B --> C[Alignment]
    C --> D[Quantification]
    D --> E[Count Matrix]
    E --> F[Normalization]
    F --> G{Analysis Branch}
    G --> H[Clustering]
    G --> I[Differential Expression]
    H --> J[Cell Type Annotation]
    I --> K[Pathway Analysis]
    J --> L[Visualization]
    K --> L
    
    style A fill:#e1f5ff
    style L fill:#ffe1e1

Loading

Example Mermaid for RQ-Dataset mapping:

graph LR
    D1[Dataset 1: scRNA-seq] --> RQ1[RQ1: TME Composition]
    D2[Dataset 2: Visium] --> RQ1
    D2 --> RQ2[RQ2: Spatial Patterns]
    D3[Dataset 3: Public Ref] --> RQ1
    
    RQ1 --> A1[Clustering]
    RQ1 --> A2[DE Analysis]
    RQ2 --> A3[Spatial Stats]
    
    style D1 fill:#c3e6cb
    style D2 fill:#c3e6cb
    style D3 fill:#c3e6cb
    style RQ1 fill:#fff3cd
    style RQ2 fill:#fff3cd

Loading

GitHub rendering: ✅ Automatic (wrap in ```mermaid code blocks)

Approach 2: GraphViz DOT Language

Pros:

• Powerful layout algorithms
• Fine-grained control
• Publication-quality output

Cons:

• Not directly rendered by GitHub (requires external tool or image)
• More complex syntax

Example DOT syntax:

digraph analysis_pipeline {
    rankdir=TB;
    node [shape=box, style=rounded];
    
    raw [label="Raw Data"];
    qc [label="QC"];
    norm [label="Normalization"];
    cluster [label="Clustering"];
    de [label="DE Analysis"];
    viz [label="Visualization"];
    
    raw -> qc;
    qc -> norm;
    norm -> cluster;
    norm -> de;
    cluster -> viz;
    de -> viz;
    
    {rank=same; cluster; de}
}

Rendering options:

• Generate PNG/SVG via dot command
• Use online tools (GraphvizOnline)
• GitHub Actions to auto-generate images

Approach 3: ASCII/Unicode Text Diagrams

Pros:

• No external tools needed
• Works everywhere (plain text)
• Fast to create

Cons:

• Less visually appealing
• Limited layout options
• Manual alignment tedious

Example ASCII:

        ┌─────────┐
        │Raw Data │
        └────┬────┘
             │
             ▼
        ┌─────────┐
        │   QC    │
        └────┬────┘
             │
             ▼
        ┌─────────┐
        │  Norm   │
        └────┬────┘
             │
        ┌────┴────┐
        │         │
        ▼         ▼
   ┌─────────┐ ┌─────────┐
   │Cluster  │ │   DE    │
   └────┬────┘ └────┬────┘
        │           │
        └─────┬─────┘
              │
              ▼
        ┌─────────┐
        │   Viz   │
        └─────────┘

Tools to help:

• boxes (CLI tool for ASCII boxes)
• Online ASCII diagram tools
• Manual creation

Approach 4: PlantUML

Pros:

• Multiple diagram types (activity, sequence, class)
• Simpler syntax than GraphViz
• Many rendering options

Cons:

• Requires PlantUML server or local install
• Not natively rendered by GitHub

Example PlantUML:

@startuml
start
:Raw Data;
:Quality Control;
:Normalization;
fork
  :Clustering;
fork again
  :Differential Expression;
end fork
:Visualization;
stop
@enduml

Approach 5: D3.js / Observable Notebooks

Pros:

• Highly interactive
• Beautiful, customizable
• Can embed in web exports

Cons:

• Requires JavaScript/HTML
• Not directly in markdown
• More complex to generate

Use case: Advanced interactive dashboards for stakeholder presentations

Implementation Strategy

Phase 1: Mermaid-Based Basic Graphs

Deliverable: Auto-generate Mermaid diagrams from specs

Implementation:

1. Parse intent.md for analysis objectives and dependencies
2. Parse relationships.md for RQ-Dataset mappings
3. Generate Mermaid flowchart syntax
4. Append to project/ directory as _analysis_dag.md and _rq_dataset_map.md
5. Include in main project README with links

Command: /biospec.generate_graphs

Example output file structure:

project/
├── _analysis_dag.md
│   └── Contains Mermaid flowchart of analysis pipeline
├── _rq_dataset_map.md
│   └── Contains Mermaid graph of RQ-Dataset-Analysis links
└── _resource_timeline.md
    └── Contains Mermaid Gantt chart (if milestones with dates)

Phase 2: Enhanced with Metadata

Add to graphs:

• Node colors based on type (data=green, analysis=blue, output=red)
• Edge labels showing data types
• Estimated time/resources per node
• Status indicators (complete/in-progress/pending)

Example enhanced Mermaid:

graph TD
    A[Dataset 1: scRNA-seq<br/>50 samples, 10GB]:::dataset
    B[QC<br/>~2 hrs, 16GB RAM]:::analysis
    C[Normalization<br/>~1 hr, 8GB RAM]:::analysis
    D[Clustering<br/>~3 hrs, 32GB RAM]:::analysis
    
    A -->|H5AD| B
    B -->|Filtered H5AD| C
    C -->|Normalized| D
    
    classDef dataset fill:#c3e6cb,stroke:#28a745
    classDef analysis fill:#cce5ff,stroke:#007bff

Loading

Phase 3: Interactive Features (Advanced)

For web exports:

• Click nodes to see details
• Filter graph by RQ or dataset
• Show/hide analysis branches
• Zoom and pan
• Highlight critical path

Technology: D3.js or similar

Phase 4: Live Updates

During project execution:

• Update graphs as analyses complete
• Add status badges to nodes (✅ done, 🔄 running, ⬜ pending)
• Show current bottlenecks
• Update with actual vs estimated resources

Graph Generation Rules

From intent.md Dependencies

Text format:

## Dependencies
- Cell type annotation must be completed before differential expression
- InferCNV requires cell annotations
- Batch correction must precede integration

Parsing logic:

1. Extract dependency statements
2. Identify "X before Y" or "Y requires X" patterns
3. Create directed edges: X → Y
4. Build DAG ensuring no cycles

Edge cases:

• Circular dependencies → Flag error
• Missing dependencies → Infer from common workflow patterns
• Parallel analyses → Place at same rank

From intent.md Milestones

Text format:

## Milestones
- Milestone 1: Data acquisition and QC (Week 1-2)
- Milestone 2: Preprocessing and normalization (Week 3-4)
- Milestone 3: Analysis and interpretation (Week 5-8)

Parsing logic:

1. Extract milestone sequence
2. Parse activities within each milestone
3. Create grouped nodes
4. Connect milestones sequentially
5. Extract timelines if present for Gantt chart

From relationships.md Mappings

Text format:

### RQ1-TME: What is the composition of the tumor microenvironment?

**Relevant Datasets**:
- Dataset 1 (Melanoma scRNA-seq): Provides cell-type resolution
- Dataset 2 (Visium spatial): Provides spatial context

Parsing logic:

1. Extract RQ identifier and title
2. Extract associated datasets
3. Extract associated analyses (from same section or intent.md)
4. Create tripartite graph: Dataset → RQ → Analysis

From datasets.md Processing Levels

Text format:

## Dataset 1: Melanoma Samples
- Primary Data Type: FASTQ (primary)
- Supplementary Data: Count matrix (secondary), Annotated object (tertiary)

Parsing logic:

1. Extract data types and processing levels
2. Infer transformation steps: primary → secondary → tertiary
3. Create data flow edges with processing methods

Visualization Quality Guidelines

Node Design

• Datasets: Green rounded rectangles
• Research Questions: Yellow hexagons
• Analyses: Blue rectangles
• Outputs: Red rounded rectangles
• Milestones: Gray containers/clusters

Edge Design

• Sequential: Solid arrows
• Optional/Alternative: Dashed arrows
• Data flow: Labeled with data type
• Dependencies: Bold arrows

Layout Principles

• Top-to-bottom: For pipelines (raw data at top, outputs at bottom)
• Left-to-right: For RQ-Dataset mappings
• Grouped: Related analyses in same visual cluster

Accessibility

• Color-blind safe palette
• Text labels on all nodes
• Alt text for generated images
• ASCII fallback option

Integration Points

With Project Templates

• Auto-generate graphs when templates validated (>80% complete)
• Update graphs when templates modified
• Include graph generation in /biospec.create_project workflow

With Validation Tool

• Check that dependencies form valid DAG (no cycles)
• Flag orphaned nodes (unconnected analyses)
• Validate that all RQs map to at least one dataset

With Review Tool

• Include graph in review output
• Annotate graph with potential issues
• Highlight resource bottlenecks visually

With GitHub Workflow

• Auto-regenerate graphs on spec updates (GitHub Actions)
• Include graphs in PR descriptions
• Track graph evolution over project lifetime

Technical Implementation

File Structure

project/
├── _graphs/
│   ├── analysis_pipeline.mermaid
│   ├── analysis_pipeline.png (optional pre-rendered)
│   ├── rq_dataset_map.mermaid
│   ├── data_flow.mermaid
│   └── resource_timeline.mermaid (Gantt)
└── README.md (includes graphs)

Command Interface

# Generate all graphs
/biospec.generate_graphs

# Generate specific graph type
/biospec.generate_graphs --type pipeline
/biospec.generate_graphs --type mapping
/biospec.generate_graphs --type dataflow

# Update existing graphs
/biospec.update_graphs

# Export to image format
/biospec.generate_graphs --export png

Code Structure

# Pseudo-code structure

class GraphGenerator:
    def __init__(self, project_path):
        self.templates = load_templates(project_path)
    
    def parse_dependencies(self):
        """Extract dependencies from intent.md"""
        return dependency_list
    
    def parse_rq_dataset_mapping(self):
        """Extract mappings from relationships.md"""
        return mappings
    
    def build_dag(self, dependencies):
        """Build directed acyclic graph"""
        return graph_structure
    
    def validate_dag(self, dag):
        """Check for cycles, orphans"""
        return validation_results
    
    def generate_mermaid(self, dag, graph_type):
        """Convert to Mermaid syntax"""
        return mermaid_code
    
    def write_graph_file(self, mermaid_code, filename):
        """Write to markdown file"""
        pass

Dependency Parsing Patterns

Pattern matching for dependencies:

• "X before Y" → X → Y
• "Y requires X" → X → Y
• "Y depends on X" → X → Y
• "After X, then Y" → X → Y
• "X and Y can run in parallel" → X, Y at same rank

NLP approach (advanced):

• Use spaCy or similar for dependency parsing
• Extract subject-verb-object relationships
• Identify causal/temporal relationships

Example Outputs

Complete Analysis Pipeline DAG

graph TD
    START([Project Start])
    D1[Dataset 1: scRNA-seq<br/>50 patients]
    D2[Dataset 2: Visium<br/>10 patients]
    
    START --> D1
    START --> D2
    
    D1 --> QC1[QC: scRNA-seq<br/>Filter cells, Remove doublets]
    D2 --> QC2[QC: Spatial<br/>Tissue detection]
    
    QC1 --> NORM1[Normalization<br/>SCTransform]
    QC2 --> NORM2[Normalization<br/>Spatial method]
    
    NORM1 --> CLUST[Clustering<br/>Leiden algorithm]
    CLUST --> ANNO[Cell Type Annotation<br/>Reference-based]
    
    ANNO --> DE[Differential Expression<br/>MAST]
    ANNO --> INFERCNV[InferCNV<br/>Malignant scoring]
    
    NORM2 --> SPATIAL[Spatial Analysis<br/>Moran's I]
    
    DE --> PATH[Pathway Analysis<br/>GSEA]
    INFERCNV --> INT[Integration<br/>scRNA + Spatial]
    SPATIAL --> INT
    
    PATH --> VIZ[Visualization<br/>Figures]
    INT --> VIZ
    
    VIZ --> END([Manuscript])
    
    style START fill:#f9f9f9
    style END fill:#f9f9f9
    style D1 fill:#c3e6cb
    style D2 fill:#c3e6cb
    style QC1 fill:#cce5ff
    style QC2 fill:#cce5ff
    style CLUST fill:#fff3cd
    style VIZ fill:#ffe1e1

Loading

RQ-Dataset-Analysis Network

graph LR
    subgraph Datasets
        D1[Dataset 1<br/>scRNA-seq]
        D2[Dataset 2<br/>Visium]
        D3[Dataset 3<br/>Public Ref]
    end
    
    subgraph Research Questions
        RQ1[RQ1: TME<br/>Composition]
        RQ2[RQ2: Spatial<br/>Patterns]
        RQ3[RQ3: Malignant<br/>Features]
    end
    
    subgraph Analyses
        A1[Clustering]
        A2[DE]
        A3[Spatial Stats]
        A4[InferCNV]
        A5[Reference<br/>Mapping]
    end
    
    D1 --> RQ1
    D1 --> RQ3
    D2 --> RQ2
    D2 --> RQ1
    D3 --> RQ1
    
    RQ1 --> A1
    RQ1 --> A2
    RQ2 --> A3
    RQ3 --> A4
    RQ1 --> A5
    
    style D1 fill:#c3e6cb
    style D2 fill:#c3e6cb
    style D3 fill:#c3e6cb
    style RQ1 fill:#fff3cd
    style RQ2 fill:#fff3cd
    style RQ3 fill:#fff3cd

Loading

Edge Cases & Considerations

Complex Branching

• Conditional analyses (if X then Y else Z)
• Optional analyses (nice-to-have)
• Alternative methods (Method A or Method B)

Representation: Dashed lines for optional/alternative paths

Iterative Processes

• QC → fail → re-run with different parameters
• Clustering → validation → re-cluster

Representation: Feedback loops with dashed arrows, or note as iterative

Parallel Workflows

• Multiple independent analysis branches
• Multi-dataset parallel processing

Representation: Place at same rank/level

Large Complex Projects

• 10+ datasets, 20+ analyses
• Graph becomes unreadable

Solutions:

• Generate multiple focused graphs (per RQ, per dataset)
• Hierarchical graphs (high-level → detailed)
• Interactive filtering

Success Metrics

Useful visualization if:

• Identifies bottlenecks or critical path
• Reveals missing dependencies
• Clarifies project structure for new team members
• Supports stakeholder communication
• Helps plan parallel vs sequential execution
• Tracks project progress visually

Related Issues

• Implement document linkages between spec entities #3 Implement document linkages between spec entities
• Programmatic template validation and cross-reference checking #12 Programmatic template validation and cross-reference checking
• Split templates into granular subfiles #4 Split templates into granular subfiles (affects graph granularity)

Priority

Medium: Would significantly improve project understanding and communication, but not blocking for core functionality.

Implementation order:

1. Design graph types and Mermaid templates
2. Implement parsing for dependencies and relationships
3. Generate basic DAG from intent.md
4. Generate RQ-Dataset map from relationships.md
5. Add styling and metadata to graphs
6. Integrate with validation and review tools
7. Add advanced features (interactive, status updates)