- Auron has passed all TPC-DS/TPC-H benchmark cases. Comparing to Spark-3.5, Auron is running ~2x faster and save ~50% cluster resources.
+ Apache Auron (Incubating) has passed all TPC-DS/TPC-H benchmark cases. Comparing to Apache Spark-3.5, Apache Auron (Incubating) is running ~2x faster and save ~50% cluster resources.
See
Benchmark Details.
diff --git a/docs/archives/v6.0.0-incubating.md b/docs/archives/v6.0.0-incubating.md
index 784e58f..ee2405e 100644
--- a/docs/archives/v6.0.0-incubating.md
+++ b/docs/archives/v6.0.0-incubating.md
@@ -11,7 +11,7 @@ title: 6.0.0-incubating
## Improvements
* Stability: Improved handling of stage retry on shuffle failures and memory spilling.
-* Modularity: Restructured codebase by extracting Celeborn, Uniffle, and Paimon into separate 3rdparty modules.
+* Modularity: Restructured codebase by extracting Celeborn, Uniffle, and Apache Paimon into separate 3rdparty modules.
* Observability: Improved logging with Thread IDs and enhanced Spark UI metrics for skew detection.
* Uniffle Integration: Improved support and documentation for Uniffle shuffle manager.
* Minor Performance Improvement: Optimized batch serde, array interleavig and coalescing.
diff --git a/docs/archives/v7.0.0-incubating.md b/docs/archives/v7.0.0-incubating.md
index 0663a34..dc7cb9c 100644
--- a/docs/archives/v7.0.0-incubating.md
+++ b/docs/archives/v7.0.0-incubating.md
@@ -5,7 +5,7 @@ title: 7.0.0-incubating
## New Features
* **AuronConfiguration**: Introduced new configuration system with dynamic default values
* **New Modules**: Added auron-core, auron-flink-extension, auron-memmgr, auron-iceberg, and auron-spark-tests modules for better modularity
-* **Spark UI Integration**: Introduced comprehensive Spark UI support with build info display and metrics
+* **Spark UI Integration**: Introduced comprehensive Apache Spark UI support with build info display and metrics
* **Extended Native Functions**: Implemented native functions for round, pow/power, lpad, rpad, reverse, initcap, levenshtein, quarter, hour, minute, second, find_in_set, nvl, nvl2, least, greatest, isnan, bround
* **Enhanced Expressions**: Native support for EqualNullSafe expression, CollectLimit, monotonically_increasing_id(), and spark_partition_id()
* **Columnar Aggregation**: Implemented columnar aggregate buffers for improved performance
diff --git a/docs/documents/benchmarks.md b/docs/documents/benchmarks.md
index 1783284..641e158 100644
--- a/docs/documents/benchmarks.md
+++ b/docs/documents/benchmarks.md
@@ -6,11 +6,11 @@ title: Benchmarks
## Benchmark Result
-Here is the benchmark result of TPC-DS 1TB Dataset, running under Spark-3.5.6 and Auron-6.0.0-preview (dc8d7a9).
+Here is the benchmark result of TPC-DS 1TB Dataset, running under Apache Spark-3.5.6 and Apache Auron (Incubating)-6.0.0-preview (dc8d7a9).
-Below is a brief introduction of how we run TPC-DS benchmark with Spark/Auron.
+Below is a brief introduction of how we run TPC-DS benchmark with Apache Spark/Apache Auron (Incubating).
## Get TPC-DS tools
```
@@ -47,8 +47,8 @@ cd tpcds/benchmark-runner
mvn package -DskipTests
```
-Edit your `$SPARK_HOME/conf/spark-default.conf` to enable/disable Auron (see the following conf), then launch benchmark runner.
-If benchmarking with Auron, ensure that the Auron jar package is correctly built and moved into `$SPARK_HOME/jars`. ([How to build Auron?](https://github.com/apache/auron/#build-from-source))
+Edit your `$SPARK_HOME/conf/spark-default.conf` to enable/disable Apache Auron (Incubating) (see the following conf), then launch benchmark runner.
+If benchmarking with Apache Auron (Incubating), ensure that the Apache Auron (Incubating) jar package is correctly built and moved into `$SPARK_HOME/jars`. ([How to build Apache Auron (Incubating)?](https://github.com/apache/auron/#build-from-source))
```bash
# use correct SPARK_HOME and data location
SPARK_HOME=$HOME/software/spark ./bin/run \
diff --git a/docs/documents/configuration.md b/docs/documents/configuration.md
index 29e4fb6..65f7d8e 100644
--- a/docs/documents/configuration.md
+++ b/docs/documents/configuration.md
@@ -2,7 +2,7 @@
title: Configurations
---
-# Configurations for Auron
+# Configurations for Apache Auron (Incubating)
### Runtime Configuration
| Conf Key | Type | Default Value | Description |
diff --git a/docs/documents/getting-started.md b/docs/documents/getting-started.md
index 7dfd630..a5fb103 100644
--- a/docs/documents/getting-started.md
+++ b/docs/documents/getting-started.md
@@ -2,21 +2,21 @@
title: Getting-Started
---
-# Getting Started with Auron for Apache Spark
+# Getting Started with Apache Auron (Incubating) for Apache Spark
## Build from source
-To build Auron from source, follow the steps below:
+To build Apache Auron (Incubating) from source, follow the steps below:
1. Install Rust
-Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling.
+Apache Auron (Incubating)'s native execution lib is written in Rust. You need to install Rust (nightly) before compiling.
We recommend using [rustup](https://rustup.rs/) for installation.
2. Install JDK
-Auron has been well tested with JDK 8, 11, and 17.
+Apache Auron (Incubating) has been well tested with JDK 8, 11, and 17.
Make sure `JAVA_HOME` is properly set and points to your desired version.
@@ -24,18 +24,18 @@ Make sure `JAVA_HOME` is properly set and points to your desired version.
4. Build the project.
-You can build Auron either *locally* or *inside Docker* using one of the supported OS images via the unified script: `auron-build.sh`.
+You can build Apache Auron (Incubating) either *locally* or *inside Docker* using one of the supported OS images via the unified script: `auron-build.sh`.
Run `./auron-build.sh --help` to see all available options.
After the build completes, a fat JAR with all dependencies will be generated in either the `target/` directory (for local builds)
or `target-docker/` directory (for Docker builds), depending on the selected build mode.
-## Run Spark Job with Auron Accelerator
+## Run Apache Spark Job with Apache Auron (Incubating) Accelerator
-This section describes how to submit and configure a Spark Job with Auron support.
+This section describes how to submit and configure an Apache Spark Job with Apache Auron (Incubating) support.
-1. Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).
+1. Move the Apache Auron (Incubating) JAR to the Apache Spark client classpath (normally spark-xx.xx.xx/jars/).
2. Add the following configs to spark configuration in `spark-xx.xx.xx/conf/spark-default.conf`:
@@ -54,15 +54,15 @@ spark.executor.memoryOverhead 4096
```shell
spark-sql -f tpcds/q01.sql
```
-## Integrating Auron with Remote Shuffle Services (RSS)
+## Integrating Apache Auron (Incubating) with Remote Shuffle Services (RSS)
-Auron can be integrated with external Remote Shuffle Services to enhance shuffle performance and improve scalability.
+Apache Auron (Incubating) can be integrated with external Remote Shuffle Services to enhance shuffle performance and improve scalability.
Currently, `Apache Celeborn` and `Apache Uniffle` are supported. Please run `./auron-build.sh --help` for the up-to-date supported versions.
### Apache Celeborn
-Auron can work with Celeborn as a shuffle manager. Integration involves configuring Auron/Spark to use the AuronCelebornShuffleManager and pointing it to the appropriate Celeborn master endpoints and storage locations. This allows Spark jobs running on Auron to leverage Celeborn for distributed shuffling.
+Apache Auron (Incubating) can work with Celeborn as a shuffle manager. Integration involves configuring Apache Auron (Incubating)/Apache Spark to use the AuronCelebornShuffleManager and pointing it to the appropriate Celeborn master endpoints and storage locations. This allows Apache Spark jobs running on Apache Auron (Incubating) to leverage Celeborn for distributed shuffling.
You can integrate using the following example configuration:
@@ -76,7 +76,7 @@ spark.sql.adaptive.localShuffleReader.enabled false
### Apache Uniffle
-Similarly, Auron also supports Uniffle, you need to configure Auron/Spark to use the AuronUniffleShuffleManager and specify the Uniffle coordinator endpoints.
+Similarly, Apache Auron (Incubating) also supports Uniffle, you need to configure Apache Auron (Incubating)/Apache Spark to use the AuronUniffleShuffleManager and specify the Uniffle coordinator endpoints.
You can integrate using the following example configuration:
@@ -88,7 +88,7 @@ spark.rss.enabled true
```
### Notes
-1. Ensure the relevant RSS client JARs are included in your Spark application's classpath.
+1. Ensure the relevant RSS client JARs are included in your Apache Spark application's classpath.
2. Replace endpoints and directories with the actual addresses in your cluster.
diff --git a/docs/introduction.md b/docs/introduction.md
index c50b789..255d160 100644
--- a/docs/introduction.md
+++ b/docs/introduction.md
@@ -3,7 +3,7 @@ title: Introduction
---
## Overview
-The [Auron](https://github.com/apache/auron) accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing framework.
+The [Apache Auron (Incubating)](https://github.com/apache/auron) accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing framework.
## Introduction
@@ -11,24 +11,24 @@ The [Auron](https://github.com/apache/auron) accelerator for Apache Spark levera
### Problem Statement
Apache Spark is a popular distributed computing framework for handling large-scale data processing tasks. However, as the data size increases, traditional row-based processing can lead to significant CPU latencies and become a performance bottleneck. To overcome this challenge, vectorized execution technology has been introduced as an optimization method for Spark.
-### Auron’s Solution
-Vectorized execution technology operates by processing data in batches rather than rows, reducing function calls and improving computation performance with SIMD instructions. Auron leverages this technology by integrating the Apache Arrow-DataFusion library with the Spark framework.
+### Apache Auron (Incubating)’s Solution
+Vectorized execution technology operates by processing data in batches rather than rows, reducing function calls and improving computation performance with SIMD instructions. Apache Auron (Incubating) leverages this technology by integrating the Apache Arrow-DataFusion library with the Spark framework.
-Auron checks and translates supported operators in the Spark's physical plan and generates an equivalent native execution plan, then it passes the generated execution plan to the underlying native engine through JNI calls. The native engine executes the plan with DataFusion framework, which benefits from vectorized execution and has better performance comparing to Spark's JVM based execution.
+Apache Auron (Incubating) checks and translates supported operators in the Spark’s physical plan and generates an equivalent native execution plan, then it passes the generated execution plan to the underlying native engine through JNI calls. The native engine executes the plan with DataFusion framework, which benefits from vectorized execution and has better performance comparing to Spark’s JVM based execution.
### Target User
-Auron's target users are those who want to accelerate Spark SQL/DataFrame queries. Users can install Auron as a Spark client extension. After installing, most SQL queries should run faster without modifying, and save cluster resources.
+Apache Auron (Incubating)’s target users are those who want to accelerate Spark SQL/DataFrame queries. Users can install Apache Auron (Incubating) as a Spark client extension. After installing, most SQL queries should run faster without modifying, and save cluster resources.
## Architecture
-The architecture design of Auron is as follows.
-Auron takes a fully optimized physical plan from Spark, mapping it into equivalent execution plan implemented in native engine, and executes in Spark distributed environment.
+The architecture design of Apache Auron (Incubating) is as follows.
+Apache Auron (Incubating) takes a fully optimized physical plan from Spark, mapping it into equivalent execution plan implemented in native engine, and executes in Spark distributed environment.
-
+
-Auron is composed of the following high-level components:
+Apache Auron (Incubating) is composed of the following high-level components:
- **Spark Extension**: hooks the whole accelerator into Spark execution lifetime.
-- **Spark Shims**: specialized codes for different versions of spark.
+- **Spark Shims**: specialized codes for different versions of Spark.
- **Native Engine**: implements the native engine in rust, including:
- ExecutionPlan protobuf specification.
- JNI gateway.
@@ -37,13 +37,13 @@ Auron is composed of the following high-level components:
The architecture diagram of the **native engine** is as follows:
-
+
### Currently Supported Native Operators/Expressions
-All supported operators in Auron are listed below. Auron does support fallbacking an operator to spark execution which has not been implemented, so SQLs containing unsupported operators can still be executed successfully. However, fallbacks takes extra costs, too many fallbacks will slow down the execution.
+All supported operators in Apache Auron (Incubating) are listed below. Apache Auron (Incubating) does support fallbacking an operator to Spark execution which has not been implemented, so SQLs containing unsupported operators can still be executed successfully. However, fallbacks takes extra costs, too many fallbacks will slow down the execution.
-Most spark builtin expressions are supported in Auron (by translating to DataFusion-physical-exprs). Auron also supports expression-level fallbacking, which can fallback a single unsupported expression to spark execution. so SQLs containing some unsupported expressions like UDF/UDTFs can still be optimized.
+Most Spark builtin expressions are supported in Apache Auron (Incubating) (by translating to DataFusion-physical-exprs). Apache Auron (Incubating) also supports expression-level fallbacking, which can fallback a single unsupported expression to Spark execution. so SQLs containing some unsupported expressions like UDF/UDTFs can still be optimized.
@@ -140,11 +140,11 @@ Most spark builtin expressions are supported in Auron (by translating to DataFus
## Join the Community
### Source Code
-Please see [Auron source code](https://github.com/apache/auron) for more information.
+Please see [Apache Auron (Incubating) source code](https://github.com/apache/auron) for more information.
### Cooperators
-Auron currently has some users and contributors:
+Apache Auron (Incubating) currently has some users and contributors: