Commits of plugin engine-datafusion

[sandeshkr419]

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[gemini-code-assist]

Summary of Changes

Hello @sandeshkr419, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant infrastructure for vectorized execution and columnar storage within OpenSearch. It adds a new parquet-data-format module, leveraging Rust via JNI for efficient Parquet file writing, and a vectorized-exec-spi library for common interfaces. Additionally, it integrates a new engine-datafusion plugin to utilize DataFusion for query execution, complete with cache management and runtime monitoring. The changes also include necessary build system updates, JVM configurations for native memory, and modifications to field mappers to support a pluggable data format.

Highlights

• New Vectorized Execution Infrastructure: Introduced a new parquet-data-format module with Rust JNI integration for efficient Parquet file writing and a vectorized-exec-spi library for common vectorized execution interfaces.
• DataFusion Plugin Integration: Added a new engine-datafusion plugin that leverages Rust JNI bindings to integrate the DataFusion query engine, complete with cache management and runtime monitoring capabilities.
• Build System and JVM Configuration Updates: Modified .gitattributes and .gitignore to support Parquet files and Rust build artifacts. Updated jvm.options and opensearch.yml to include native memory tracking and DataFusion-specific memory pool settings. Gradle build scripts were also updated to include new modules and remote plugin loading.
• Pluggable Data Format Feature for Mappers: Modified ScaledFloatFieldMapper.java and TokenCountFieldMapper.java to conditionally use a new pluggable data format feature, indicating a move towards more flexible data handling.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/datafusion-e2e-test.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant amount of new functionality by adding the engine-datafusion and parquet-data-format plugins. The changes are extensive, touching build configurations, adding new SPIs, and implementing JNI bridges to Rust code for high-performance data processing. While this is a substantial contribution, the PR in its current state has several critical issues that need to be addressed. Multiple checks (API annotations, JarHell, Javadocs) and test assertions are disabled, which is risky. There are also several bugs, such as incorrect enum values, wrong return types, and potential build-breaking configurations in the Rust code. The practice of adding new classes to dependency packages (e.g., org.apache.lucene) is also a major concern. It's recommended to address these fundamental issues and re-enable the disabled checks to ensure the stability and maintainability of the codebase.

[gemini-code-assist]

The CSV enum constant is initialized with the string "parquet". This is likely a copy-paste error and will cause incorrect behavior when working with CSV data formats. It should be initialized with "csv".

Suggested change

	CSV("parquet"),
	CSV("csv"),

[gemini-code-assist]

This file and others in this pull request are being added to the org.apache.lucene.* package. Adding classes to a package owned by a third-party dependency is a strong anti-pattern. It can lead to classpath issues, split-package problems with the Java Platform Module System (JPMS), and unexpected behavior if the dependency is updated. All new classes should reside within the project's own namespace, such as org.opensearch.*.

[gemini-code-assist]

The JNI functions in this file (e.g., Java_com_parquet_parquetdataformat_bridge_RustBridge_createWriter) return a jint to indicate success or failure. However, the corresponding Java native methods in RustBridge.java are declared as void and throws IOException. This is a signature mismatch. The Rust code should throw a Java exception using env.throw_new() on failure instead of returning an integer code. The current implementation will silently swallow errors on the Java side.

[gemini-code-assist]

The Rust edition is set to "2024". The 2024 edition of Rust is not yet stable and is only available on nightly toolchains. This will cause build failures for anyone using a stable Rust compiler. Please use the latest stable edition, which is 2021.

Suggested change

	edition = "2024"
	edition = "2021"

[gemini-code-assist]

The entire processing loop for API annotations (@PublicApi, @ExperimentalApi, @DeprecatedApi) has been commented out. This effectively disables all API compatibility and visibility checks, which is critical for maintaining a stable public API for OpenSearch. This should be re-enabled and any resulting errors should be fixed.

Suggested change

	// for (var element : elements) {
	// validate(element);
	//
	// if (!checkPackage(element)) {
	// continue;
	// }
	//
	// // Skip all not-public elements
	// checkPublicVisibility(null, element);
	//
	// if (element instanceof TypeElement) {
	// process((TypeElement) element);
	// }
	// }
	for (var element : elements) {
	validate(element);
	if (!checkPackage(element)) {
	continue;
	}
	// Skip all not-public elements
	checkPublicVisibility(null, element);
	if (element instanceof TypeElement) {
	process((TypeElement) element);
	}
	}

[gemini-code-assist]

The logic to find the cargo executable by checking a list of hardcoded paths is brittle and may fail on systems where cargo is installed in a different location. It's better to rely on cargo being available in the system's PATH, which is the standard practice. This simplifies the build script and makes it more portable.

[gemini-code-assist]

This method indexDataToParquetEngine appears to be for local testing. It's commented out, but it should be removed entirely from production code before merging.

[gemini-code-assist]

Several new projects are being excluded from Javadoc checks. While this might be acceptable during initial development, it's important to add Javadocs for public APIs before the feature is finalized to ensure maintainability. The TODO comments suggest this is temporary, which is good, but this should be tracked for completion.

[gemini-code-assist]

The -Darrow.memory.debug.allocator=true option is being enabled by default. This is a debugging flag for Apache Arrow that can have a significant performance impact. It should not be enabled by default in a production configuration. Consider removing it or leaving it commented out with an explanation.

[gemini-code-assist]

The line to add a settings update consumer for the cache size limit is commented out. This means that if the cache size limit is changed dynamically via the cluster settings API, the change will not be propagated to the native cache manager. This functionality should be enabled to allow for dynamic cache configuration.