Squashed data fusion changes by sandeshkr419 · Pull Request #270 · sandeshkr419/OpenSearch

sandeshkr419 · 2025-12-01T04:45:17Z

Description

Squashing all commits in my private repo to cherry-pick easily

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Marc Handalian <marc.handalian@gmail.com> (cherry picked from commit cb75910)

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit eb01905)

…project#19399) Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit c9d5b17)

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit 98de93e)

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit 5fef617)

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit e4ebf59)

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Co-authored-by: Arpit Bandejiya <abandeji@amazon.com> Signed-off-by: bharath-techie <bharath78910@gmail.com>

Co-authored-by: Arpit Bandejiya <abandeji@amazon.com> Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

* Abstracting lucene away: part 1 * initial abstractions to reduce indexing engine coupling * Text backed engine testing --------- Co-authored-by: Mohit Godwani <mgodwan@amazon.com>

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

…roject#19441) Signed-off-by: expani <anijainc@amazon.com>

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

* Changes in dataformat for CSVEngine Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * Changes for Reader to work Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> --------- Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> Co-authored-by: Bharathwaj G <bharath78910@gmail.com> Signed-off-by: bharath-techie <bharath78910@gmail.com>

Signed-off-by: bharath-techie <bharath78910@gmail.com>

* Integrate aggregators to convert result from datafusion Signed-off-by: expani <anijainc@amazon.com> * Initialised bigArrays and queryCollManagers for DatafusionContext Signed-off-by: expani <anijainc@amazon.com> * Refactored to set agg result within utility Signed-off-by: expani <anijainc@amazon.com> --------- Signed-off-by: expani <anijainc@amazon.com>

Co-authored-by: Arpit Bandejiya <abandeji@amazon.com> Signed-off-by: bharath-techie <bharath78910@gmail.com>

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Signed-off-by: expani <anijainc@amazon.com>

* Feature/datafusion 4 (#46) * Composite document writer pool initial implementation * Committer interface and lucene based commit engine implementation * Catalog snapshot changes to create segment view during commit --------- Co-authored-by: Shashank Gowri <shnkgo@amazon.com> * fix build for commit integration Signed-off-by: bharath-techie <bharath78910@gmail.com> --------- Signed-off-by: bharath-techie <bharath78910@gmail.com> Co-authored-by: Shashank Gowri <shnkgo@amazon.com>

* Add memory pool changes for datafusion Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * Add comments over the new monitor structs Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * Add changes for wrapper of global runTimeEnv Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * Add documentation related setting and make the memory pool static * comment the print for memory * Fix tests Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> --------- Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Signed-off-by: expani <anijainc@amazon.com>

…0029)

* Add parquet benchmarks in missing javadoc Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * add flag to disable java doc check --------- Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

* Single commit with all changes * Update the e2e test workflow * Ignore lint * Add missing doc flag * fix newline * remove extra missingdoc flag in action * Add changes to have all queries * Run only DatafusionReaderManager test Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * Fix 200 status code Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> --------- Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

* Add CacheManager # Conflicts: # plugins/engine-datafusion/jni/src/lib.rs # plugins/engine-datafusion/jni/src/listing_table.rs # plugins/engine-datafusion/jni/src/row_id_optimizer.rs # plugins/engine-datafusion/jni/src/util.rs # Conflicts: # plugins/engine-datafusion/jni/src/lib.rs # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/DataFusionQueryJNI.java # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/DataFusionService.java # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/core/GlobalRuntimeEnv.java # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/search/DatafusionSearcher.java # plugins/engine-datafusion/src/test/java/org/opensearch/datafusion/DataFusionReaderManagerTests.java # Conflicts: # plugins/engine-datafusion/jni/src/lib.rs # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/DataFusionPlugin.java # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/DataFusionService.java # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/core/GlobalRuntimeEnv.java # plugins/engine-datafusion/src/main/java/org/opensearch/datafusion/search/DatafusionSearcher.java # plugins/engine-datafusion/src/test/java/org/opensearch/datafusion/DataFusionReaderManagerTests.java # plugins/engine-datafusion/src/test/java/org/opensearch/datafusion/DataFusionServiceTests.java # plugins/engine-datafusion/src/test/java/org/opensearch/datafusion/TestDataFusionServiceTests.java * create CacheManager struct * rebase * add file2 * fix references * cleanup commented code * remove unused references * rebase1 * Add Native Handles * moving cacheOperations to use runtimeEnv * fix references * runtimeHandle reference * rename configPtr reference

…#20048) * add stream and allocator closure post fetch phase Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Move stream and allocator for executeQueryPhase into correct scope Signed-off-by: Marc Handalian <marc.handalian@gmail.com> --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

…nsearch-project#20032) * Injecting count sub agg if absent to ensure doc_count is correct Signed-off-by: expani <anijainc@amazon.com> * Sorting keys for merge sort to work Signed-off-by: expani <anijainc@amazon.com> * Went into zen mode and made code simpler Signed-off-by: expani <anijainc@amazon.com> --------- Signed-off-by: expani <anijainc@amazon.com>

)

…ject#20041) * Single commit with all changes * Update the e2e test workflow * Clean unused functions Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * remove imports --------- Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

…ct#20051)

* fixing path reference while adding to cache * remove unnecessary logs

…earch-project#20060)

…roject#20070) Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

…s with Searcher (opensearch-project#20063) * CatalogSnapshot Manager + postDelete hooks * Fix tests * Minor fixes to post delete hooks * Fix IndexFileDeleterTests * Add IndexFileDeleter tests in E2E * Empty commit to re-run E2E tests

…0074)

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

…project#20079) These tests run 42 or 43 queries right now and match results for correctness. all queries flagged as supported are expected to succeed for this check to pass Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

…sync changes (opensearch-project#20071) * Changes for tokio io/cpu runtime + search flow async changes Signed-off-by: bharath-techie <bharath78910@gmail.com> * minor fixes and todos Signed-off-by: bharath-techie <bharath78910@gmail.com> * Addressing comment to move to ActionListener Signed-off-by: bharath-techie <bharath78910@gmail.com> * addressing comments Signed-off-by: bharath-techie <bharath78910@gmail.com> * fix to sort numeric aggs Signed-off-by: bharath-techie <bharath78910@gmail.com> --------- Signed-off-by: bharath-techie <bharath78910@gmail.com>

…ect#20110) Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

…pensearch-project#20119) Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

…pensearch-project#19978) * Adding JNI abstractions for logging in rust through RustLoggerBridge Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> * Adding logging for java in parquet-data-format module and cleaning up stale files Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com> --------- Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>

* Refactor stream handling for DF execution * Change to Buffer Allocator for DF execution * Introduce RecordBatchIterator and integrate with collector * Minor fixup * Update VSRManager to update field vector atomically * Fixup * Fixup * Fixup * Test fixup * Integrate with AsyncRecordBatchIterator * Fix up

gemini-code-assist · 2025-12-01T04:47:09Z

Summary of Changes

Hello @sandeshkr419, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances OpenSearch's data processing capabilities by integrating the DataFusion query engine and introducing native support for the Parquet columnar data format. It lays the groundwork for a vectorized execution framework, enabling more efficient storage and querying of analytical workloads. The changes involve new modules for a vectorized execution Service Provider Interface (SPI), a dedicated Parquet data format plugin with a Rust backend, and a DataFusion engine plugin that orchestrates these components for advanced query execution, complete with robust memory management and caching.

Highlights

DataFusion Integration: Introduces a new engine-datafusion plugin that integrates the DataFusion query engine into OpenSearch, enabling advanced vectorized query execution capabilities.
Parquet Data Format Support: Adds a new parquet-data-format module, allowing OpenSearch to store and query data in the columnar Parquet format, leveraging a high-performance native Rust backend via JNI.
Vectorized Execution SPI: Establishes a new opensearch-vectorized-exec-spi library, defining core interfaces and utilities for pluggable vectorized execution engines and data formats within OpenSearch.
Native Rust Backend: Both the Parquet data format and DataFusion engine utilize Rust-based JNI libraries for low-level, high-performance operations, including file writing, merging, and query execution.
Memory Management and Caching: Implements sophisticated memory management for Apache Arrow buffers and introduces caching mechanisms for file metadata within the DataFusion engine to optimize query performance.
Pluggable Data Format Architecture: Modifies existing mappers (ScaledFloatFieldMapper, TokenCountFieldMapper) to support a new pluggable data format architecture, allowing dynamic integration of formats like Parquet.

Ignored Files

Ignored by pattern: .github/workflows/** (1)
- .github/workflows/datafusion-e2e-test.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a significant new feature for data fusion, including support for the Parquet data format, a vectorized execution SPI, and a DataFusion engine plugin. The changes span across Java, Gradle, and Rust code, setting up the foundation for columnar data processing.

While the overall direction is promising, my review has identified several critical issues that must be addressed before this can be considered for merging. These include disabled assertions in tests, commented-out API validation logic, a memory leak in the native resource handling, and several build-breaking issues in the Rust code such as an invalid edition and JNI signature mismatches. There are also some bugs in the Java code, like incorrect enum values. These issues suggest that the code is in a very early stage of development and requires substantial refinement to meet production quality standards.

gemini-code-assist · 2025-12-01T04:50:12Z

+                // TODO : remove this - this disables assertions
+                node.jvmArgs(" -da ");


This change disables assertions (-da) for test cluster nodes. The TODO comment indicates this might be temporary, but merging this would cause tests to run without critical checks, potentially masking bugs and regressions. Assertions should be enabled in test environments to ensure code correctness. Please remove this line before merging.

gemini-code-assist · 2025-12-01T04:50:12Z

+//        for (var element : elements) {
+//            validate(element);
+//
+//            if (!checkPackage(element)) {
+//                continue;
+//            }
+//
+//            // Skip all not-public elements
+//            checkPublicVisibility(null, element);
+//
+//            if (element instanceof TypeElement) {
+//                process((TypeElement) element);
+//            }
+//        }


The core logic of the ApiAnnotationProcessor has been commented out. This processor is essential for enforcing API backward compatibility and proper usage of annotations like @PublicApi and @ExperimentalApi. Disabling it, even temporarily, is dangerous as it can lead to unintended breaking changes. This code must be restored, and any compilation issues that led to it being commented out should be fixed properly.

gemini-code-assist · 2025-12-01T04:50:12Z

+            throw new IllegalArgumentException("Null native pointer");
+        }
+        this.ptr = ptr;
+        this.cleanable = CLEANER.register(this, new CleanupAction(ptr, this::doClose));


The use of java.lang.ref.Cleaner here will cause a memory leak. The Runnable passed to CLEANER.register is created with a method reference this::doClose, which captures a strong reference to the NativeHandle instance (this). According to the Cleaner documentation, the cleanup action must not hold a strong reference to the object being registered, otherwise the object will never become phantom-reachable and the cleanup action will never run. This means both the Java object and the native memory it points to will be leaked.

The CleanupAction should be a static class that does not hold a reference to the NativeHandle instance. It should operate on the native pointer (ptr) and use a static method for deallocation.

gemini-code-assist · 2025-12-01T04:50:12Z

+@ExperimentalApi
+public enum DataFormat {
+    /** CSV Format*/
+    CSV("parquet"),


The CSV enum constant is incorrectly initialized with the string "parquet". This will lead to incorrect data format identification. It should be initialized with "csv".

Suggested change

CSV("parquet"),

CSV("csv"),

gemini-code-assist · 2025-12-01T04:50:12Z

+    }
+
+    public DataFormat getDataFormat() {
+        return DataFormat.CSV;


The getDataFormat() method in ParquetDataSourceCodec incorrectly returns DataFormat.CSV. It should return DataFormat.PARQUET to correctly identify itself as the codec for the Parquet format.

Suggested change

return DataFormat.CSV;

return DataFormat.PARQUET;

gemini-code-assist · 2025-12-01T04:50:13Z

+[package]
+name = "rust"
+version = "0.1.0"
+edition = "2024"


The Rust edition is set to 2024, which is not yet a stable edition. This will cause the build to fail with current stable Rust toolchains. Please use the latest stable edition, which is 2021.

Suggested change

edition = "2024"

edition = "2021"

gemini-code-assist · 2025-12-01T04:50:13Z

+pub extern "system" fn Java_com_parquet_parquetdataformat_bridge_RustBridge_createWriter(
+    mut env: JNIEnv,
+    _class: JClass,
+    file: JString,
+    schema_address: jlong
+) -> jint {
+    let filename: String = env.get_string(&file).expect("Couldn't get java string!").into();
+    match NativeParquetWriter::create_writer(filename, schema_address as i64) {
+        Ok(_) => 0,
+        Err(_) => -1,
+    }
+}
+
+#[unsafe(no_mangle)]
+pub extern "system" fn Java_com_parquet_parquetdataformat_bridge_RustBridge_write(
+    mut env: JNIEnv,
+    _class: JClass,
+    file: JString,
+    array_address: jlong,
+    schema_address: jlong
+) -> jint {
+    let filename: String = env.get_string(&file).expect("Couldn't get java string!").into();
+    match NativeParquetWriter::write_data(filename, array_address as i64, schema_address as i64) {
+        Ok(_) => 0,
+        Err(_) => -1,
+    }
+}
+
+#[unsafe(no_mangle)]
+pub extern "system" fn Java_com_parquet_parquetdataformat_bridge_RustBridge_closeWriter(
+    mut env: JNIEnv,
+    _class: JClass,
+    file: JString
+) -> jint {
+    let filename: String = env.get_string(&file).expect("Couldn't get java string!").into();
+    match NativeParquetWriter::close_writer(filename) {
+        Ok(_) => 0,
+        Err(_) => -1,
+    }
+}
+
+#[unsafe(no_mangle)]
+pub extern "system" fn Java_com_parquet_parquetdataformat_bridge_RustBridge_flushToDisk(
+    mut env: JNIEnv,
+    _class: JClass,
+    file: JString
+) -> jint {
+    let filename: String = env.get_string(&file).expect("Couldn't get java string!").into();
+    match NativeParquetWriter::flush_to_disk(filename) {
+        Ok(_) => 0,
+        Err(_) => -1,
+    }


There is a JNI signature mismatch between the Java native method declarations in RustBridge.java and their Rust implementations here. The Java methods are declared as void and throws IOException, while the Rust JNI functions are implemented to return a jint. This will cause an UnsatisfiedLinkError at runtime.

The Rust functions should be updated to have a void return type and to throw a Java exception when an error occurs, to match the Java declaration. For example, instead of returning -1, use env.throw_new("java/io/IOException", error_message).

gemini-code-assist · 2025-12-01T04:50:13Z

 compileJava {
-  options.compilerArgs += ['-processor', ['org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor',
-    'org.opensearch.common.annotation.processor.ApiAnnotationProcessor'].join(',')]
+  options.compilerArgs += ['-processor', ['org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor'].join(',')]


The ApiAnnotationProcessor has been removed from the list of annotation processors for the server project. This is a critical change that disables API compatibility checks. This processor is vital for maintaining API stability and should not be removed. This change should be reverted.

gemini-code-assist · 2025-12-01T04:50:13Z

+/modules/parquet-data-format/jni/target/.rustc_info.json
+
+/modules/parquet-data-format/jni/target/release
+**/Cargo.lock


The pattern **/Cargo.lock ignores all Cargo.lock files throughout the repository. For applications and top-level library crates, Cargo.lock should be checked into version control to ensure reproducible builds. Globally ignoring it can lead to dependency hell and non-deterministic builds across different environments. Please consider removing this global ignore and only ignore Cargo.lock files within build artifact directories (like target/), if necessary.

gemini-code-assist · 2025-12-01T04:50:13Z

+-XX:+UnlockDiagnosticVMOptions
+
+# Enabling debug logs for Allocators in Arrow
+-Darrow.memory.debug.allocator=true


Enabling debug logs for the Arrow allocator (-Darrow.memory.debug.allocator=true) in the default jvm.options is not recommended for production builds, as it can introduce performance overhead. This setting is useful for development and debugging but should be disabled by default in a distribution.

mch2 and others added 30 commits November 4, 2025 10:20

Initial commit of plugin engine-datafusion

0140eff

Signed-off-by: Marc Handalian <marc.handalian@gmail.com> (cherry picked from commit cb75910)

Fixed the build failures, added javadocs (opensearch-project#19398)

b7c5cfb

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit eb01905)

Add JNI layer and rust methods to execute substrait plan (opensearch-…

e700056

…project#19399) Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit c9d5b17)

Add License, make precommit checks work

90d5676

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit 98de93e)

Ensure precommit succeeds

073afec

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit 5fef617)

Added Integration of Search with Datafusion plugin

2284dcd

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> (cherry picked from commit e4ebf59)

Add extensions for csv codec

1eaca09

Signed-off-by: bharath-techie <bharath78910@gmail.com>

adding libs , data source plugin and data source aware plugin

a1e4ae6

Signed-off-by: bharath-techie <bharath78910@gmail.com>

search interface changes

a739841

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Search and indexing engine integration changes - inprogesss commit

e17bce1

Signed-off-by: bharath-techie <bharath78910@gmail.com>

in-progress read engine / query phase abstractions

f2dc283

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Tying searcher and reader with rust

3508b17

Co-authored-by: Arpit Bandejiya <abandeji@amazon.com> Signed-off-by: bharath-techie <bharath78910@gmail.com>

Add changes for searcher integration

72e75f1

Co-authored-by: Arpit Bandejiya <abandeji@amazon.com> Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Fix datafusion rust

17c2a2c

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

added global row id optimizer and tests for query phase

7a935b2

Feature/datafusion (#38)

7698cf3

* Abstracting lucene away: part 1 * initial abstractions to reduce indexing engine coupling * Text backed engine testing --------- Co-authored-by: Mohit Godwani <mgodwan@amazon.com>

Fixing tests

947291d

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Fix Listing Cache

a146b3e

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Integrate aggregators to convert result from datafusion (opensearch-p…

546ef45

…roject#19441) Signed-off-by: expani <anijainc@amazon.com>

Changes in dataformat for CSVEngine

fffbc30

Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Changes to make plugin contexts work with source parse

3accadb

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Indexing integration

a8ddfcb

Co-authored-by: Arpit Bandejiya <abandeji@amazon.com> Signed-off-by: bharath-techie <bharath78910@gmail.com>

removing CSV codec and integrating with parquet module

9f1eed6

Signed-off-by: bharath-techie <bharath78910@gmail.com>

fixes for publishToMavenLocal

802b335

Signed-off-by: bharath-techie <bharath78910@gmail.com>

add readme

ca6283c

Signed-off-by: bharath-techie <bharath78910@gmail.com>

Added support for average metric aggregation (opensearch-project#19559)

33fb888

Signed-off-by: expani <anijainc@amazon.com>

Fixing rust build for parquet-data-format (opensearch-project#19611)

3e074dd

alchemist51 and others added 25 commits November 17, 2025 22:12

Aligned platform directory with build script (opensearch-project#20028)

2ca0f80

Signed-off-by: expani <anijainc@amazon.com>

Changing path to match gradle native output dir (opensearch-project#2…

46b7228

…0029)

Add parquet benchmarks in missing javadoc (opensearch-project#20043)

a844c24

* Add parquet benchmarks in missing javadoc Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> * add flag to disable java doc check --------- Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Preserving order of Segment in CatalogSnapshot (opensearch-project#20045

9f94eef

)

Support log byte merge policy (opensearch-project#20053)

a85f603

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Close searcher on context close and fix unit tests. (opensearch-proje…

b1a1d30

…ct#20051)

Fix path reference while adding to cache (opensearch-project#20052)

f5a7030

* fixing path reference while adding to cache * remove unnecessary logs

Don't create catalog snapshot in refresh if no changes in file (opens…

e980a19

…earch-project#20060)

Consume same writer generation count in merge and write (opensearch-p…

bda4e93

…roject#20070) Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Handle LocalDateTime for min and max aggregator (opensearch-project#2…

20fe987

…0074)

partial aggs changes for distinct count (opensearch-project#20008)

cf5fad4

Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>

disabling row ID optimizer (opensearch-project#20086)

6d744bf

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Support debug mode for parquet and datafusion plugin (opensearch-proj…

23e4717

…ect#20110) Signed-off-by: Arpit Bandejiya <abandeji@amazon.com>

Refactoring merge code to act at composite level instead of Parquet (o…

a786542

…pensearch-project#20119) Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

gemini-code-assist Bot reviewed Dec 1, 2025

View reviewed changes

sandeshkr419 merged commit 0154013 into 3.3-df Dec 1, 2025
12 of 14 checks passed

sandeshkr419 deleted the feature/datafusion branch December 1, 2025 04:53

sandeshkr419 restored the feature/datafusion branch December 1, 2025 04:56

Navigation Menu

Squashed data fusion changes - #270

Squashed data fusion changes#270
sandeshkr419 merged 111 commits into
3.3-dffrom
feature/datafusion

sandeshkr419 commented Dec 1, 2025

Uh oh!

gemini-code-assist Bot commented Dec 1, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

		// TODO : remove this - this disables assertions
		node.jvmArgs(" -da ");

Conversation

sandeshkr419 commented Dec 1, 2025

Description

Related Issues

Check List

Uh oh!

gemini-code-assist Bot commented Dec 1, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants