Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .asf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# limitations under the License.

github:
description: "Apache Auron Site"
description: "Apache Auron (Incubating) Site"
homepage: https://auron.apache.org/
labels:
- auron
Expand Down
4 changes: 2 additions & 2 deletions docs/.vuepress/components/BenchmarkChart.vue
Original file line number Diff line number Diff line change
Expand Up @@ -111,15 +111,15 @@ const updateChartOption = () => {
},
series: [
{
name: 'Spark-3.5.1',
name: 'Apache Spark-3.5.1',
type: 'bar',
barWidth: config.barWidth,
data: sparkResultValues,
markLine: {
data: [{ type: 'average', name: 'Average' }],
},
}, {
name: 'Auron-6.0.0-preview',
name: 'Apache Auron (Incubating)-6.0.0-preview',
type: 'bar',
barWidth: config.barWidth,
data: blazeResultValues,
Expand Down
2 changes: 1 addition & 1 deletion docs/.vuepress/components/Footer.vue
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
</div>
<div class="footer-copyright">
<br>
<p>Apache Auron is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p>
<p>Apache Auron (Incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p>
<p>Copyright © 2025-2026 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.</p>
<p>Apache, the names of Apache projects, and the feather logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
</div>
Expand Down
14 changes: 7 additions & 7 deletions docs/.vuepress/components/Head.vue
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<template>
<div class="home-banner">
<div class="banner-title-container">
<div class="banner-title">Run Spark SQL/DataFrame Faster</div>
<div class="banner-subtitle">The Auron accelerator for big data engine (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing.</div>
<div class="banner-title">Apache Auron (Incubating)</div>
<div class="banner-subtitle">The accelerator for big data engines (e.g., Apache Spark, Apache Flink) leveraging native vectorized execution to accelerate query processing.</div>
<div class="github-buttons-container"><GitHubBadge /></div>
<a class="route-link auto-link vp-hero-action-button primary" href="/documents/getting-started" aria-label="Get Started">Get Started</a>
</div>
Expand Down Expand Up @@ -44,7 +44,7 @@
Easy to Use
</div>
<ul class="feature-details">
<li>Simple to build and install to Spark.</li>
<li>Simple to build and install to Apache Spark.</li>
<li>Easy to configuration.</li>
<li>Full-featured execution metrics.</li>
</ul>
Expand All @@ -55,7 +55,7 @@
Compatibility
</div>
<ul class="feature-details">
<li>Adapted to Spark mainline versions.</li>
<li>Adapted to Apache Spark mainline versions.</li>
<li>Supports different storage systems like HDFS, S3, etc.</li>
</ul>
</div>
Expand All @@ -65,7 +65,7 @@
Ecosystem
</div>
<ul class="feature-details">
<li>Supports data lake system like Hudi, Paimon.</li>
<li>Supports data lake systems like Apache Hudi, Apache Paimon.</li>
<li>Supports Remote Shuffle Service like Apache Celeborn.</li>
</ul>
</div>
Expand All @@ -75,8 +75,8 @@
Community
</div>
<ul class="feature-details">
<li>Some cooperators have applied Auron on production.</li>
<li>More are researching and evaluating Auron.</li>
<li>Some cooperators have applied Apache Auron (Incubating) on production.</li>
<li>More are researching and evaluating Apache Auron (Incubating).</li>
</ul>
</div>
</div>
Expand Down
1 change: 1 addition & 0 deletions docs/.vuepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ export default defineUserConfig({
head: [
['link', { rel: 'icon', href: '/logo-mini.png' }]
],
title: 'Apache Auron (Incubating)',
theme: defaultTheme({
logo: '/auron.svg',
home: '/',
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ home: true
<div class="home-para">
<div class="para-title">Benchmarks</div>
<div class="para-text">
Auron has passed all TPC-DS/TPC-H benchmark cases. Comparing to Spark-3.5, Auron is running ~2x faster and save ~50% cluster resources.
Apache Auron (Incubating) has passed all TPC-DS/TPC-H benchmark cases. Comparing to Apache Spark-3.5, Apache Auron (Incubating) is running ~2x faster and save ~50% cluster resources.
See <a href="documents/benchmarks">Benchmark Details</a>.
</div>
<div class="home-benchmark-container-wrapper">
Expand Down
2 changes: 1 addition & 1 deletion docs/archives/v6.0.0-incubating.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ title: 6.0.0-incubating

## Improvements
* Stability: Improved handling of stage retry on shuffle failures and memory spilling.
* Modularity: Restructured codebase by extracting Celeborn, Uniffle, and Paimon into separate 3rdparty modules.
* Modularity: Restructured codebase by extracting Celeborn, Uniffle, and Apache Paimon into separate 3rdparty modules.
* Observability: Improved logging with Thread IDs and enhanced Spark UI metrics for skew detection.
* Uniffle Integration: Improved support and documentation for Uniffle shuffle manager.
* Minor Performance Improvement: Optimized batch serde, array interleavig and coalescing.
Expand Down
2 changes: 1 addition & 1 deletion docs/archives/v7.0.0-incubating.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ title: 7.0.0-incubating
## New Features
* **AuronConfiguration**: Introduced new configuration system with dynamic default values
* **New Modules**: Added auron-core, auron-flink-extension, auron-memmgr, auron-iceberg, and auron-spark-tests modules for better modularity
* **Spark UI Integration**: Introduced comprehensive Spark UI support with build info display and metrics
* **Spark UI Integration**: Introduced comprehensive Apache Spark UI support with build info display and metrics
* **Extended Native Functions**: Implemented native functions for round, pow/power, lpad, rpad, reverse, initcap, levenshtein, quarter, hour, minute, second, find_in_set, nvl, nvl2, least, greatest, isnan, bround
* **Enhanced Expressions**: Native support for EqualNullSafe expression, CollectLimit, monotonically_increasing_id(), and spark_partition_id()
* **Columnar Aggregation**: Implemented columnar aggregate buffers for improved performance
Expand Down
8 changes: 4 additions & 4 deletions docs/documents/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ title: Benchmarks

## Benchmark Result

Here is the benchmark result of TPC-DS 1TB Dataset, running under Spark-3.5.6 and Auron-6.0.0-preview (dc8d7a9).
Here is the benchmark result of TPC-DS 1TB Dataset, running under Apache Spark-3.5.6 and Apache Auron (Incubating)-6.0.0-preview (dc8d7a9).

<BenchmarkChart />

Below is a brief introduction of how we run TPC-DS benchmark with Spark/Auron.
Below is a brief introduction of how we run TPC-DS benchmark with Apache Spark/Apache Auron (Incubating).

## Get TPC-DS tools
```
Expand Down Expand Up @@ -47,8 +47,8 @@ cd tpcds/benchmark-runner
mvn package -DskipTests
```

Edit your `$SPARK_HOME/conf/spark-default.conf` to enable/disable Auron (see the following conf), then launch benchmark runner.
If benchmarking with Auron, ensure that the Auron jar package is correctly built and moved into `$SPARK_HOME/jars`. ([How to build Auron?](https://github.com/apache/auron/#build-from-source))
Edit your `$SPARK_HOME/conf/spark-default.conf` to enable/disable Apache Auron (Incubating) (see the following conf), then launch benchmark runner.
If benchmarking with Apache Auron (Incubating), ensure that the Apache Auron (Incubating) jar package is correctly built and moved into `$SPARK_HOME/jars`. ([How to build Apache Auron (Incubating)?](https://github.com/apache/auron/#build-from-source))
```bash
# use correct SPARK_HOME and data location
SPARK_HOME=$HOME/software/spark ./bin/run \
Expand Down
2 changes: 1 addition & 1 deletion docs/documents/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Configurations
---

# Configurations for Auron
# Configurations for Apache Auron (Incubating)

### Runtime Configuration
| Conf Key | Type | Default Value | Description |
Expand Down
26 changes: 13 additions & 13 deletions docs/documents/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,40 +2,40 @@
title: Getting-Started
---

# Getting Started with Auron for Apache Spark
# Getting Started with Apache Auron (Incubating) for Apache Spark

## Build from source

To build Auron from source, follow the steps below:
To build Apache Auron (Incubating) from source, follow the steps below:

1. Install Rust

Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling.
Apache Auron (Incubating)'s native execution lib is written in Rust. You need to install Rust (nightly) before compiling.

We recommend using [rustup](https://rustup.rs/) for installation.

2. Install JDK

Auron has been well tested with JDK 8, 11, and 17.
Apache Auron (Incubating) has been well tested with JDK 8, 11, and 17.

Make sure `JAVA_HOME` is properly set and points to your desired version.

3. Check out the source code.

4. Build the project.

You can build Auron either *locally* or *inside Docker* using one of the supported OS images via the unified script: `auron-build.sh`.
You can build Apache Auron (Incubating) either *locally* or *inside Docker* using one of the supported OS images via the unified script: `auron-build.sh`.

Run `./auron-build.sh --help` to see all available options.

After the build completes, a fat JAR with all dependencies will be generated in either the `target/` directory (for local builds)
or `target-docker/` directory (for Docker builds), depending on the selected build mode.

## Run Spark Job with Auron Accelerator
## Run Apache Spark Job with Apache Auron (Incubating) Accelerator

This section describes how to submit and configure a Spark Job with Auron support.
This section describes how to submit and configure an Apache Spark Job with Apache Auron (Incubating) support.

1. Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).
1. Move the Apache Auron (Incubating) JAR to the Apache Spark client classpath (normally spark-xx.xx.xx/jars/).

2. Add the following configs to spark configuration in `spark-xx.xx.xx/conf/spark-default.conf`:

Expand All @@ -54,15 +54,15 @@ spark.executor.memoryOverhead 4096
```shell
spark-sql -f tpcds/q01.sql
```
## Integrating Auron with Remote Shuffle Services (RSS)
## Integrating Apache Auron (Incubating) with Remote Shuffle Services (RSS)

Auron can be integrated with external Remote Shuffle Services to enhance shuffle performance and improve scalability.
Apache Auron (Incubating) can be integrated with external Remote Shuffle Services to enhance shuffle performance and improve scalability.

Currently, `Apache Celeborn` and `Apache Uniffle` are supported. Please run `./auron-build.sh --help` for the up-to-date supported versions.

### Apache Celeborn

Auron can work with Celeborn as a shuffle manager. Integration involves configuring Auron/Spark to use the AuronCelebornShuffleManager and pointing it to the appropriate Celeborn master endpoints and storage locations. This allows Spark jobs running on Auron to leverage Celeborn for distributed shuffling.
Apache Auron (Incubating) can work with Celeborn as a shuffle manager. Integration involves configuring Apache Auron (Incubating)/Apache Spark to use the AuronCelebornShuffleManager and pointing it to the appropriate Celeborn master endpoints and storage locations. This allows Apache Spark jobs running on Apache Auron (Incubating) to leverage Celeborn for distributed shuffling.

You can integrate using the following example configuration:

Expand All @@ -76,7 +76,7 @@ spark.sql.adaptive.localShuffleReader.enabled false

### Apache Uniffle

Similarly, Auron also supports Uniffle, you need to configure Auron/Spark to use the AuronUniffleShuffleManager and specify the Uniffle coordinator endpoints.
Similarly, Apache Auron (Incubating) also supports Uniffle, you need to configure Apache Auron (Incubating)/Apache Spark to use the AuronUniffleShuffleManager and specify the Uniffle coordinator endpoints.

You can integrate using the following example configuration:

Expand All @@ -88,7 +88,7 @@ spark.rss.enabled true
```

### Notes
1. Ensure the relevant RSS client JARs are included in your Spark application's classpath.
1. Ensure the relevant RSS client JARs are included in your Apache Spark application's classpath.

2. Replace endpoints and directories with the actual addresses in your cluster.

Expand Down
30 changes: 15 additions & 15 deletions docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,32 @@ title: Introduction
---

## Overview
The [Auron](https://github.com/apache/auron) accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing framework.
The [Apache Auron (Incubating)](https://github.com/apache/auron) accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing framework.


## Introduction

### Problem Statement
Apache Spark is a popular distributed computing framework for handling large-scale data processing tasks. However, as the data size increases, traditional row-based processing can lead to significant CPU latencies and become a performance bottleneck. To overcome this challenge, vectorized execution technology has been introduced as an optimization method for Spark.

### Auron’s Solution
Vectorized execution technology operates by processing data in batches rather than rows, reducing function calls and improving computation performance with SIMD instructions. Auron leverages this technology by integrating the Apache Arrow-DataFusion library with the Spark framework.
### Apache Auron (Incubating)’s Solution
Vectorized execution technology operates by processing data in batches rather than rows, reducing function calls and improving computation performance with SIMD instructions. Apache Auron (Incubating) leverages this technology by integrating the Apache Arrow-DataFusion library with the Spark framework.

Auron checks and translates supported operators in the Spark's physical plan and generates an equivalent native execution plan, then it passes the generated execution plan to the underlying native engine through JNI calls. The native engine executes the plan with DataFusion framework, which benefits from vectorized execution and has better performance comparing to Spark's JVM based execution.
Apache Auron (Incubating) checks and translates supported operators in the Sparks physical plan and generates an equivalent native execution plan, then it passes the generated execution plan to the underlying native engine through JNI calls. The native engine executes the plan with DataFusion framework, which benefits from vectorized execution and has better performance comparing to Sparks JVM based execution.

### Target User
Auron's target users are those who want to accelerate Spark SQL/DataFrame queries. Users can install Auron as a Spark client extension. After installing, most SQL queries should run faster without modifying, and save cluster resources.
Apache Auron (Incubating)’s target users are those who want to accelerate Spark SQL/DataFrame queries. Users can install Apache Auron (Incubating) as a Spark client extension. After installing, most SQL queries should run faster without modifying, and save cluster resources.

## Architecture
The architecture design of Auron is as follows.
Auron takes a fully optimized physical plan from Spark, mapping it into equivalent execution plan implemented in native engine, and executes in Spark distributed environment.
The architecture design of Apache Auron (Incubating) is as follows.
Apache Auron (Incubating) takes a fully optimized physical plan from Spark, mapping it into equivalent execution plan implemented in native engine, and executes in Spark distributed environment.

![Spark+Auron architecture](./img/auron_architecture.webp)
![Apache Spark + Apache Auron (Incubating) architecture](./img/auron_architecture.webp)

Auron is composed of the following high-level components:
Apache Auron (Incubating) is composed of the following high-level components:

- **Spark Extension**: hooks the whole accelerator into Spark execution lifetime.
- **Spark Shims**: specialized codes for different versions of spark.
- **Spark Shims**: specialized codes for different versions of Spark.
- **Native Engine**: implements the native engine in rust, including:
- ExecutionPlan protobuf specification.
- JNI gateway.
Expand All @@ -37,13 +37,13 @@ Auron is composed of the following high-level components:

The architecture diagram of the **native engine** is as follows:

![Auron Native Engine](./img/auron_native_engine.webp)
![Apache Auron (Incubating) Native Engine](./img/auron_native_engine.webp)

### Currently Supported Native Operators/Expressions

All supported operators in Auron are listed below. Auron does support fallbacking an operator to spark execution which has not been implemented, so SQLs containing unsupported operators can still be executed successfully. However, fallbacks takes extra costs, too many fallbacks will slow down the execution.
All supported operators in Apache Auron (Incubating) are listed below. Apache Auron (Incubating) does support fallbacking an operator to Spark execution which has not been implemented, so SQLs containing unsupported operators can still be executed successfully. However, fallbacks takes extra costs, too many fallbacks will slow down the execution.

Most spark builtin expressions are supported in Auron (by translating to DataFusion-physical-exprs). Auron also supports expression-level fallbacking, which can fallback a single unsupported expression to spark execution. so SQLs containing some unsupported expressions like UDF/UDTFs can still be optimized.
Most Spark builtin expressions are supported in Apache Auron (Incubating) (by translating to DataFusion-physical-exprs). Apache Auron (Incubating) also supports expression-level fallbacking, which can fallback a single unsupported expression to Spark execution. so SQLs containing some unsupported expressions like UDF/UDTFs can still be optimized.

<table class="my-table3">
<tbody>
Expand Down Expand Up @@ -140,11 +140,11 @@ Most spark builtin expressions are supported in Auron (by translating to DataFus
## Join the Community

### Source Code
Please see [Auron source code](https://github.com/apache/auron) for more information.
Please see [Apache Auron (Incubating) source code](https://github.com/apache/auron) for more information.

### Cooperators

Auron currently has some users and contributors:
Apache Auron (Incubating) currently has some users and contributors:

<div class="partners-container">
<div class="partners">
Expand Down