Skip to content

Repository files navigation

RNA-seq Analysis Pipeline

This repository contains RNA-seq analysis workflows for MAPK/ERK pathway inhibitor studies in lung cancer cell lines. Features publication-quality GSEA visualizations and transcription factor family enrichment analysis.

Projects

Project Cell Line Treatments Description
8442 PC9 (EGFR-mutant NSCLC) DMSO, Gefitinib (EGFR-i), SCH772984 (ERK-i) EGFR vs ERK inhibition
8446 A549 (KRAS-mutant NSCLC) DMSO, LY300 (RAF-i), SCH772984 (ERK-i) RAF vs ERK inhibition

Key Features

  • Publication-quality GSEA dot plots with viridis color scale
  • Four pathway databases: Hallmark, KEGG, Reactome, GO:BP
  • Comparison plots with NES-based color encoding (orangered=up, royalblue=down)
  • Combined 4-panel figures for cross-database comparison
  • Transcription Factor family enrichment using CISBP database (~60 DBD families)

Directory Structure

RNA-seq_pipeline/
├── README.md
├── 8442_PC9_analysis/           # PC9 cell line analysis
│   ├── 20251218_Set_up_files_for_nf-core.Rmd
│   ├── 20251220_RNA-seq_analysis.Rmd
│   ├── 20260120_RNA-seq_analysis.Rmd          # Updated GSEA visualizations
│   ├── 20260120_RNA-seq_analysis_TF_enrichment.Rmd  # TF family analysis
│   ├── setup_rnaseq_pipeline.sh
│   ├── run_nfcore_rnaseq_ULTRA_SAFE.sh
│   └── launch_pipeline_ultra_safe.sh
│
└── 8446_A549_analysis/          # A549 cell line analysis
    ├── 20260106_Set_up_files_for_nf-core.Rmd
    ├── 20260106_RNA-seq_analysis.Rmd
    ├── setup_rnaseq_pipeline.sh
    ├── run_nfcore_rnaseq_ULTRA_SAFE.sh
    └── launch_pipeline_ultra_safe.sh

Workflow Overview

The analysis workflow consists of two main phases:

Phase 1: Data Processing (nf-core/rnaseq)

  1. Data Ingestion (*_Set_up_files_for_nf-core.Rmd)

    • Parse FASTQ filenames to extract metadata
    • Create sample sheet for nf-core pipeline
    • Inspect file sizes and estimate read counts
    • Export metadata and sample sheet CSV
  2. Pipeline Setup (setup_rnaseq_pipeline.sh)

    • Validate prerequisites (conda, R, tidyverse)
    • Create conda environment with Nextflow
    • Generate nf-core sample sheet format
    • Create pipeline execution scripts
  3. Pipeline Execution (run_nfcore_rnaseq_ULTRA_SAFE.sh or launch_pipeline_ultra_safe.sh)

    • Run nf-core/rnaseq with STAR + Salmon
    • Ultra-safe mode: 1 STAR job at a time, 28GB RAM limit
    • Automatic resume on failure
    • ~10-12 hours runtime for 9 samples

Phase 2: Differential Expression Analysis

  1. DESeq2 Analysis (*_RNA-seq_analysis.Rmd)

    • Load counts from Salmon output
    • Quality control (PCA, sample distances, correlations)
    • Differential expression analysis
    • Volcano plots, MA plots, heatmaps
    • Gene Set Enrichment Analysis (fgsea) with 4 databases:
      • MSigDB Hallmark (50 gene sets)
      • KEGG pathways
      • Reactome pathways
      • GO Biological Process
    • Publication-quality GSEA dot plots
    • Cross-database comparison figures
  2. TF Family Enrichment (*_TF_enrichment.Rmd) - NEW

    • Transcription factor family enrichment using CISBP database
    • ~60 DNA-binding domain (DBD) families analyzed
    • Individual plots per treatment + comparison plots
    • Shows TFs with coordinated expression changes per family

Quick Start Guide

Step 1: Run Data Ingestion (Local R/RStudio)

Open the *_Set_up_files_for_nf-core.Rmd file in RStudio and knit to HTML. This will:

  • Parse FASTQ filenames
  • Create sample_sheet.csv
  • Generate metadata files

Step 2: Copy Sample Sheet to Server

# For 8442 (PC9)
scp sample_sheet.csv user@server:/DATA/usr/d.goodall/Projects/PARM/E2845_PC9_Gef30nM_Sch810nM/RNA_seq/8442_PC9_analysis/

# For 8446 (A549)
scp sample_sheet.csv user@server:/DATA/usr/d.goodall/Projects/PARM/E2848_A549_Ly300_Sch77/RNA-seq/8446_A549_analysis/

Step 3: Run Pipeline Setup (Server)

# Navigate to analysis directory
cd /DATA/usr/d.goodall/Projects/PARM/E2848_A549_Ly300_Sch77/RNA-seq/8446_A549_analysis

# Run setup script
bash setup_rnaseq_pipeline.sh

Step 3b: Make Scripts Executable (if needed)

If you get "Permission denied" errors when running scripts, make them executable:

# Make all shell scripts executable
chmod +x *.sh

# Or individually:
chmod +x launch_pipeline_ultra_safe.sh
chmod +x run_nfcore_rnaseq_ULTRA_SAFE.sh
chmod +x setup_rnaseq_pipeline.sh

Step 4: Launch Pipeline (Server)

Option A: Background execution (recommended)

bash launch_pipeline_ultra_safe.sh

Option B: Interactive execution

bash run_nfcore_rnaseq_ULTRA_SAFE.sh

Option C: SLURM submission

sbatch --cpus-per-task=16 --mem=64G --time=24:00:00 \
       --job-name=rnaseq_8446 run_nfcore_rnaseq_ULTRA_SAFE.sh

Step 5: Monitor Progress

# View log
tail -f pipeline_ultra_safe.log

# Check STAR jobs (should be 0 or 1)
ps aux | grep "STAR --genomeDir" | grep -v grep | wc -l

# Check memory usage
free -h

if need to kill the processes

1. Find your processes:

ps aux | grep $(whoami) | grep -iE "nextflow|java|star|salmon|trim_galore|fastqc|samtools|picard" | grep -v grep
2. Kill them all at once:


# Kill nextflow and its spawned Java process first
pkill -u $(whoami) -f nextflow
pkill -u $(whoami) -f "java.*nextflow"

# Kill any remaining pipeline tools
pkill -u $(whoami) -f STAR
pkill -u $(whoami) -f salmon
pkill -u $(whoami) -f trim_galore
pkill -u $(whoami) -f fastqc
pkill -u $(whoami) -f samtools
pkill -u $(whoami) -f picard

Before re-running on HPC, clean the failed task caches:

cd /DATA/shared/projects/EGFR_TFs/00_MAPK/01_PARM/03_E2896_H1975_Osi_Sch77/RNA-seq
conda activate rnaseq
nextflow clean -f

The below checks if we are good to perform a re-run

#!/bin/bash

Pre-run diagnostic check for nf-core/rnaseq re-run

ANALYSIS_DIR="/DATA/shared/projects/EGFR_TFs/00_MAPK/01_PARM/03_E2896_H1975_Osi_Sch77/RNA-seq"

echo "═══════════════════════════════════════════════════" echo " PRE-RUN DIAGNOSTIC CHECK" echo "═══════════════════════════════════════════════════" echo ""

1. Check for running processes

echo "1. RUNNING PROCESSES" NF_PROCS=$(ps aux | grep -E "nextflow|java.*nextflow" | grep -v grep | wc -l) STAR_PROCS=$(ps aux | grep "STAR" | grep -v grep | wc -l) echo " Nextflow processes: ${NF_PROCS}" echo " STAR processes: ${STAR_PROCS}" [ "$NF_PROCS" -eq 0 ] && [ "$STAR_PROCS" -eq 0 ] && echo " ✅ All clear" || echo " ❌ Kill these first!" echo ""

2. Check nextflow clean status

echo "2. NEXTFLOW WORK DIR" if [ -d "${ANALYSIS_DIR}/work" ]; then WORK_SIZE=$(du -sh "${ANALYSIS_DIR}/work" 2>/dev/null | cut -f1) WORK_DIRS=$(find "${ANALYSIS_DIR}/work" -maxdepth 2 -mindepth 2 -type d 2>/dev/null | wc -l) echo " Work dir exists: yes (${WORK_SIZE}, ${WORK_DIRS} task dirs)" echo " ⚠️ Run 'nextflow clean -f' if not done yet" else echo " Work dir: cleaned or not present" echo " ✅ Clean" fi echo ""

3. Check lock files

echo "3. LOCK FILES" if [ -f "${ANALYSIS_DIR}/.nextflow/lock" ]; then echo " ❌ Lock file exists: ${ANALYSIS_DIR}/.nextflow/lock" echo " Run: rm ${ANALYSIS_DIR}/.nextflow/lock" else echo " ✅ No lock file" fi echo ""

4. Check report files that caused overwrite error

echo "4. EXISTING REPORT FILES" REPORT="${ANALYSIS_DIR}/results_8522/pipeline_info/execution_report_ultra_safe.html" TIMELINE="${ANALYSIS_DIR}/results_8522/pipeline_info/timeline_ultra_safe.html" [ -f "$REPORT" ] && echo " Report exists: yes (will be overwritten with new config)" || echo " Report: not present" [ -f "$TIMELINE" ] && echo " Timeline exists: yes (will be overwritten with new config)" || echo " Timeline: not present" echo " ✅ OK - new config has overwrite = true" echo ""

5. Check conda env

echo "5. CONDA ENVIRONMENT" echo " Active env: ${CONDA_DEFAULT_ENV:-none}" if command -v nextflow &> /dev/null; then NF_VER=$(nextflow -version 2>&1 | grep -oP 'version \K[0-9.]+' || echo "unknown") echo " Nextflow version: ${NF_VER}" echo " ✅ Nextflow available" else echo " ❌ Nextflow not found - activate conda env first" fi echo ""

6. Check sample sheet

echo "6. SAMPLE SHEET" SS="${ANALYSIS_DIR}/nfcore_samplesheet.csv" if [ -f "$SS" ]; then SAMPLES=$(tail -n +2 "$SS" | wc -l) echo " Found: ${SS}" echo " Samples: ${SAMPLES}" echo " ✅ Ready" else echo " ❌ Not found: ${SS}" fi echo ""

7. Check cached results

echo "7. CACHED RESULTS (from previous run)" COUNT_FILE="${ANALYSIS_DIR}/results_8522/star_salmon/salmon.merged.gene_counts.tsv" [ -f "$COUNT_FILE" ] && echo " Gene counts file: exists ✅" || echo " Gene counts file: not yet" STAR_BAMS=$(find "${ANALYSIS_DIR}/results_8522/star_salmon" -name "*.bam" 2>/dev/null | wc -l) echo " STAR BAM files: ${STAR_BAMS}" echo ""

8. Memory check

echo "8. SYSTEM RESOURCES" AVAIL_GB=$(free -g | grep Mem | awk '{print $7}') echo " Available memory: ${AVAIL_GB} GB" [ "$AVAIL_GB" -ge 50 ] && echo " ✅ Sufficient (need ~50GB peak)" || echo " ⚠️ Low - need ~50GB peak" echo ""

9. Check the updated script is in place

echo "9. UPDATED RUN SCRIPT" SCRIPT="${ANALYSIS_DIR}/run_nfcore_rnaseq_ULTRA_SAFE.sh" if [ -f "$SCRIPT" ]; then if grep -q "report.overwrite" "$SCRIPT" 2>/dev/null; then echo " ✅ Script has report.overwrite fix" else echo " ❌ Script is OLD version - copy updated script from repo" fi if grep -q "max_memory = '40.GB'" "$SCRIPT" 2>/dev/null; then echo " ✅ Script has updated max_memory (40GB)" else echo " ❌ Script has old max_memory - copy updated script from repo" fi else echo " ❌ Script not found at ${SCRIPT}" fi echo ""

echo "═══════════════════════════════════════════════════" echo " DONE - Review above and fix any ❌ items" echo "═══════════════════════════════════════════════════"

Check which sample is processing

ps aux | grep "STAR --genomeDir" | grep -v grep

  1. Check STAR jobs (MOST IMPORTANT): ps aux | grep "STAR --genomeDir" | grep -v grep | wc -l

See if still running

ps aux | grep nextflow | grep -v grep

Step 6: Run DESeq2 Analysis (Local R/RStudio)

After pipeline completes (~10-12 hours), open *_RNA-seq_analysis.Rmd in RStudio and knit to HTML.

Pipeline Configuration

Resource Limits (Ultra-Safe Mode)

Process Max Forks Memory CPUs
STAR Alignment 1 28 GB 16
Picard MarkDuplicates 2 12 GB -
Salmon Quant 2 6 GB -
RSeQC 2 8 GB -

Expected Runtime

  • Per STAR job: ~45-60 minutes
  • 9 samples: ~9 hours
  • Post-processing: ~1 hour
  • Total: ~10-12 hours

Output Files

From nf-core/rnaseq

results_*/
├── star_salmon/
│   ├── salmon.merged.gene_counts.tsv    # Gene-level counts
│   ├── salmon.merged.gene_tpm.tsv       # TPM values
│   └── [sample]/                        # Per-sample Salmon output
├── multiqc/
│   └── multiqc_report.html              # QC summary
├── fastqc/                              # Raw read QC
├── trimgalore/                          # Trimmed reads
└── pipeline_info/                       # Execution reports

From DESeq2 Analysis

DESeq2_analysis/
├── PCA_plot.png
├── sample_distance_heatmap.png
├── correlation_heatmap.png
├── dispersion_plot.png
├── volcano_plot_*.png
├── MA_plots_combined.png
├── heatmap_top_DEGs.png
├── heatmap_MAPK_pathway.png
├── DESeq2_results_*.csv                 # Full results tables
│
├── # GSEA Results (CSV)
├── GSEA_Hallmark_*.csv
├── GSEA_KEGG_*.csv
├── GSEA_Reactome_*.csv
├── GSEA_GOBP_*.csv                      # NEW: GO Biological Process
│
├── # GSEA Dot Plots (PNG + PDF)
├── GSEA_Hallmark_dotplot_*.{png,pdf}    # Individual treatment plots
├── GSEA_KEGG_dotplot_*.{png,pdf}
├── GSEA_Reactome_dotplot_*.{png,pdf}
├── GSEA_GOBP_dotplot_*.{png,pdf}
│
├── # GSEA Comparison Plots
├── GSEA_Hallmark_comparison_dotplot.{png,pdf}
├── GSEA_KEGG_comparison_dotplot.{png,pdf}
├── GSEA_Reactome_comparison_dotplot.{png,pdf}
├── GSEA_GOBP_comparison_dotplot.{png,pdf}
├── GSEA_ALL_databases_comparison.{png,pdf}  # 4-panel combined figure
│
├── # TF Family Enrichment
├── TF_family_dotplot_Gefitinib.{png,pdf}
├── TF_family_dotplot_SCH772984.{png,pdf}
├── TF_family_comparison_dotplot.{png,pdf}
├── GSEA_TF_families_*.csv
│
├── # Individual TF Gene Analysis (NEW)
├── TF_individual_dotplot_Gefitinib.{png,pdf}
├── TF_individual_dotplot_SCH772984.{png,pdf}
├── TF_individual_comparison_dotplot.{png,pdf}  # 20" x 14" faceted
│
├── DEG_summary_table.csv
└── GSEA_summary_table.csv

Experimental Details

Project 8442: PC9 (EGFR-mutant)

  • Cell line: PC9 (EGFR exon 19 deletion)
  • Treatments:
    • DMSO (vehicle control)
    • Gefitinib 30 nM (EGFR inhibitor, 24h)
    • SCH772984 810 nM (ERK1/2 inhibitor, 24h)
  • Replicates: 3 biological replicates (B1, B2, B3)
  • FASTQ path: /shared/gcf/d.goodall/8442/fastq_files
  • Output path: /DATA/usr/d.goodall/Projects/PARM/E2845_PC9_Gef30nM_Sch810nM/RNA_seq/8442_PC9_analysis

Project 8446: A549 (KRAS-mutant)

  • Cell line: A549 (KRAS G12S mutation)
  • Treatments:
    • DMSO (vehicle control)
    • LY300 (RAF inhibitor)
    • SCH772984 (ERK1/2 inhibitor)
  • Replicates: 3 biological replicates (B1, B2, B3)
  • FASTQ path: /shared/gcf/d.goodall/8446/fastq_files
  • Output path: /DATA/usr/d.goodall/Projects/PARM/E2848_A549_Ly300_Sch77/RNA-seq/8446_A549_analysis

Dependencies

R Packages

# Core analysis
library(tidyverse)
library(DESeq2)
library(AnnotationDbi)
library(org.Hs.eg.db)

# Visualization
library(ggplot2)
library(ggrepel)
library(pheatmap)
library(RColorBrewer)
library(ComplexHeatmap)
library(EnhancedVolcano)
library(scales)           # For color scaling

# GSEA
library(fgsea)
library(msigdbr)
library(pathview)

# Utilities
library(biomaRt)
library(ggpubr)
library(apeglm)
library(patchwork)        # Multi-panel figures

# TF Enrichment (NEW)
library(TFutils)          # CISBP TF catalog with DBD families

Command-line Tools

  • Nextflow (via conda)
  • nf-core/rnaseq v3.17.0 (Note: requires process.shell fix for versions 3.16.1-3.18.0)
  • Singularity (recommended) or Docker for containers (more reliable than conda)

Troubleshooting

Pipeline Fails

# Check Nextflow log
tail -100 .nextflow.log

# Check for errors
grep -i error .nextflow.log

# Resume from last checkpoint
bash launch_pipeline_ultra_safe.sh  # -resume flag is included

CRLF Line Ending Errors (Windows → Linux)

If you see errors like:

/bin/bash^M: bad interpreter: No such file or directory
'\r': command not found

Cause: Scripts edited on Windows have CRLF line endings instead of Unix LF.

Fix:

# Convert all shell scripts to Unix line endings
sed -i 's/\r$//' *.sh

# Or use dos2unix if available
dos2unix *.sh

Prevention: The repository includes a .gitattributes file that enforces LF line endings for shell scripts.

process.shell Permission Denied (Exit Code 126)

If you see errors like:

.command.run: line 330: .command.run: Permission denied
Command exit status: 126
WARN: Directive `process.shell` cannot contain new-line characters

Cause: Known bug in nf-core/rnaseq versions 3.16.1, 3.17.0, and 3.18.0. The process.shell directive contains newline characters that break newer Nextflow versions.

Fix: Already applied in run_nfcore_rnaseq_ULTRA_SAFE.sh. The custom config includes:

process.shell = ['/bin/bash', '-euo', 'pipefail']

References:

Conda Package Not Found (fq, etc.)

If you see errors like:

PackagesNotFoundError: bioconda::fq==0.9.1
PackagesNotFoundError: bioconda::fq==0.12.0

Cause: The fq package is only needed when strandedness = "auto" in the sample sheet, which triggers the FQ_SUBSAMPLE step.

Fix (already applied): The setup script now uses strandedness = "reverse" instead of "auto":

mutate(strandedness = "reverse")  # Avoids fq package requirement

Quick fix if you already have a sample sheet:

sed -i 's/,auto$/,reverse/' nfcore_samplesheet.csv

Alternative solutions:

  1. Use Singularity instead of Conda (recommended for better reproducibility):

    # Edit run_nfcore_rnaseq_ULTRA_SAFE.sh
    # Change: -profile conda
    # To:     -profile singularity
  2. Use Docker instead of Conda:

    # Change: -profile conda
    # To:     -profile docker

See also: nf-core/rnaseq Issue #1555

Out of Memory (OOM)

The ultra-safe configuration limits STAR to 28GB RAM with only 1 job at a time. If still failing:

  1. Check for other processes using memory: free -h
  2. Reduce --max_memory in config
  3. Contact system admin

Disk Space Issues

If you see errors like:

No space left on device
Unable to stage foreign file: s3://ngi-igenomes/...

Check disk space:

df -h /DATA

Free up space:

# Delete work directories from previous runs (ONLY after pipeline completes!)
rm -rf work/

# Check large files in your directory
find /DATA/usr/yourusername/ -type f -size +1G -exec ls -lh {} \; 2>/dev/null | head -20

Note: The STAR index download from S3 requires ~30GB of free space. The work directory can grow to 100+ GB during execution.

Sample Sheet Issues

Ensure sample sheet format matches:

sample,fastq_1,fastq_2,strandedness
A549_dmso_B1,/path/to/R1.fastq.gz,/path/to/R2.fastq.gz,reverse

Important: Use reverse for strandedness (standard for Illumina stranded RNA-seq) to avoid the fq package issue.

GSEA Visualization Details

Individual Dot Plots

Publication-quality dot plots for each pathway database:

Aesthetic Encoding
X-axis NES (Normalized Enrichment Score)
Y-axis Pathway name (ordered by NES)
Dot size Gene count in pathway
Dot color -log10(padj) using viridis scale
  • Vertical dashed line at NES = 0
  • Top 30 pathways shown per database
  • Significance threshold: padj < 0.05

Comparison Dot Plots

Side-by-side comparison of treatments:

Aesthetic Encoding
X-axis Treatment (Gefitinib, SCH772984)
Y-axis Pathway name (clustered by NES pattern)
Dot color NES direction: orangered (up) / royalblue (down)
Dot alpha NES magnitude (0.3-1.0)
Dot size Gene count
Markers * padj ≤ 0.01, . padj ≤ 0.05

Combined 4-Panel Figure

GSEA_ALL_databases_comparison.{png,pdf} combines all four database comparisons:

┌─────────────────┬─────────────────┐
│  A. Hallmark    │  B. KEGG        │
├─────────────────┼─────────────────┤
│  C. Reactome    │  D. GO:BP       │
└─────────────────┴─────────────────┘
  • 16" × 20" portrait format
  • Panel labels (A-D) for publication reference
  • Unified title and legend

TF Family Enrichment Analysis

Uses the CISBP database via TFutils to analyze transcription factor families:

Method

  1. TF genes grouped by DNA-binding domain (DBD) family (~60 families)
  2. fgsea enrichment analysis on ranked gene lists
  3. Leading edge size = TFs actually contributing to enrichment signal

Key Parameters

Parameter Value Rationale
minSize 2 Capture small TF families
padj_cutoff 0.25 Exploratory threshold
min_tfs_changing 1 Capture individual TF changes
Size breaks 1, 2, 3, 4, 5+ Leading edge counts

TF Family Plots

  • Individual dot plots per treatment (NES on x-axis, viridis color for significance)
  • Comparison plot with - indicator for TF families not significant in one treatment
  • Same orangered/royalblue encoding as GSEA comparison plots

Individual TF Gene Plots (NEW)

Shows actual TF gene names (not grouped by family):

Feature Value
Ranking By absolute log2 fold change
Filter padj < 0.25
Display Top 100 TFs per treatment
Comparison Side-by-side faceted layout (20" × 14")

Output Files

File Description
TF_family_dotplot_*.{png,pdf} Family-level enrichment per treatment
TF_family_comparison_dotplot.{png,pdf} Side-by-side family comparison
TF_individual_dotplot_*.{png,pdf} Individual TF genes per treatment
TF_individual_comparison_dotplot.{png,pdf} Faceted individual TF comparison
GSEA_TF_families_*.csv Full enrichment results

References

Author

D.J. Goodall

License

This project is for internal research use.

About

RNA-seq pipeline using nf-core and DESeq2 analysis from EGFR / MAPK treated NSCLC also transfected with promoter focused SuRE library

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages