Skip to content

docs: expand profiling guide with JVM and async-profiler coverage#3628

Merged
andygrove merged 5 commits intoapache:mainfrom
andygrove:docs/profiling-guide
Apr 1, 2026
Merged

docs: expand profiling guide with JVM and async-profiler coverage#3628
andygrove merged 5 commits intoapache:mainfrom
andygrove:docs/profiling-guide

Conversation

@andygrove
Copy link
Copy Markdown
Member

@andygrove andygrove commented Mar 4, 2026

Summary

Add detailed profiling guide - rendered version

Changes

  • Renames profiling_native_code.md to profiling.md and expands it into a comprehensive profiling guide
  • Adds a tool comparison table (async-profiler vs JFR vs cargo-flamegraph) with guidance on when to use each
  • Adds async-profiler section: installation, attaching to running Spark apps via asprof, java agent usage, event types, output formats, platform notes
  • Adds Java Flight Recorder section: spark-submit flags, jcmd dynamic recording, viewer options, useful JFR events for Comet debugging
  • Adds Tips for Profiling Comet section: wall-clock profiling for JNI overhead, alloc profiling around Arrow FFI, isolating Rust issues, correlating JVM/native frames
  • Preserves all existing cargo bench / cargo-flamegraph content
  • Links to benchmarks/tpc/README.md for integrated benchmark profiling

Rename profiling_native_code.md to profiling.md and add sections for
async-profiler (unified JVM + native flame graphs), Java Flight Recorder,
a tool comparison table, and practical tips for profiling Comet's mixed
JVM/Rust execution.
@andygrove andygrove marked this pull request as ready for review March 4, 2026 13:40
@andygrove andygrove requested a review from mbutrovich March 4, 2026 15:17
Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove we might want to include a full working example

For example I use

TPCH Local Q4 (Profiler)
JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.5/libexec spark-submit --master="local[*]" --conf spark.driver.memory=8G --conf spark.executor.instances=1 --conf spark.executor.cores=4 --conf spark.cores.max=4 --conf spark.executor.memory=16g --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=16g --conf spark.comet.explainFallback.enabled=true --conf spark.comet.logFallbackReasons.enabled=true --conf spark.eventLog.enabled=true --jars $COMET_JAR --driver-class-path $COMET_JAR --conf spark.driver.extraClassPath=$COMET_JAR --conf spark.executor.extraClassPath=$COMET_JAR --conf spark.plugins=org.apache.spark.CometPlugin --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager --conf spark.comet.enabled=true --conf spark.comet.exec.shuffle.enableFastEncoding=true --conf spark.comet.exec.shuffle.fallbackToColumnar=true --conf spark.comet.exec.replaceSortMergeJoin=true --conf spark.comet.cast.allowIncompatible=true --conf spark.comet.scan.impl=native_iceberg_compat --conf "spark.executor.extraJavaOptions=-agentpath:/Users/xxx/Downloads/async-profiler-4.3-macos/lib/libasyncProfiler.dylib=start,event=cpu,file=profile-executor.html,tree" --conf "spark.driver.extraJavaOptions=-agentpath:/Users/xxx/Downloads/async-profiler-4.3-macos/lib/libasyncProfiler.dylib=start,event=cpu,file=profile-driver.html,tree" dev/benchmarks/tpcbench.py --name comet --benchmark tpch --data ../datafusion-benchmarks/tpch/data --queries ../datafusion-benchmarks/tpch/queries --output . --iterations 1 --query 4

And this flag helps a bit file=profile-driver.html,tree which builds an output as a tree hierarchy

### Integrated benchmark profiling

The TPC benchmark scripts in `benchmarks/tpc/` have built-in async-profiler support via
the `--async-profiler` flag. See [benchmarks/tpc/README.md](../../benchmarks/tpc/README.md)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
the `--async-profiler` flag. See [benchmarks/tpc/README.md](../../benchmarks/tpc/README.md)
the `--async-profiler` flag. See [benchmarks/tpc/README.md](../../../benchmarks/tpc/README.md)

### Integrated benchmark profiling

The TPC benchmark scripts support `--jfr` for automatic JFR recording during benchmark
runs. See [benchmarks/tpc/README.md](../../benchmarks/tpc/README.md) for details.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
runs. See [benchmarks/tpc/README.md](../../benchmarks/tpc/README.md) for details.
runs. See [benchmarks/tpc/README.md](../../../benchmarks/tpc/README.md) for details.


```shell
# Linux x64
wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz
wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz
mkdir -p /opt/async-profiler

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, /opt usually requires sudo privileges. Maybe use $HOME/opt/... ?!

harness = false
```

These benchmarks are useful when for comparing performance between releases or between feature branches and the
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
These benchmarks are useful when for comparing performance between releases or between feature branches and the
These benchmarks are useful for comparing performance between releases or between feature branches and the

Comment thread docs/source/contributor-guide/profiling.md
andygrove and others added 2 commits March 20, 2026 08:21
@andygrove andygrove merged commit ba6b3ce into apache:main Apr 1, 2026
4 checks passed
@andygrove andygrove deleted the docs/profiling-guide branch April 1, 2026 10:25
vaibhawvipul pushed a commit to vaibhawvipul/datafusion-comet that referenced this pull request Apr 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants