Commits data fusion

[sandeshkr419]

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[gemini-code-assist]

Summary of Changes

Hello @sandeshkr419, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant architectural enhancements to OpenSearch by integrating the DataFusion query engine and enabling native Parquet data format support. The changes involve creating new plugins for Parquet data handling and DataFusion integration, along with a dedicated module for vectorized execution interfaces. This foundational work sets the stage for high-performance, columnar data processing within OpenSearch, leveraging Rust for native code execution and Apache Arrow for efficient in-memory data representation. The build system has been updated to seamlessly manage these new multi-language components, and JVM configurations are adjusted to optimize memory usage for vectorized operations.

Highlights

• New Parquet Data Format Plugin: Introduced a new plugin, modules/parquet-data-format, to enable OpenSearch to store and index data in the columnar Parquet format. This plugin leverages JNI bindings to a high-performance Rust backend for efficient Parquet file generation and management.
• New DataFusion Engine Plugin: Added a new plugin, plugins/engine-datafusion, which integrates the DataFusion query engine with OpenSearch. This integration enables vectorized query execution, significantly enhancing analytical capabilities and performance by processing data in a columnar fashion.
• Vectorized Execution Service Provider Interface (SPI): A new module, libs/vectorized-exec-spi, was created to define common interfaces for vectorized execution. This includes standardized JNI native handle management and a flexible SPI for different data formats, ensuring extensibility and interoperability.
• Enhanced Build System for Rust and Java Integration: Updated various Gradle files (.gitattributes, .gitignore, gradle/missing-javadoc.gradle, gradle/run.gradle) to support the new Rust-based components and native library handling. This includes configuring Rust build processes, managing generated files, and streamlining the integration of Java and native code.
• Optimized JVM and DataFusion Configuration: Modified distribution/src/config/jvm.options and distribution/src/config/opensearch.yml to include settings for native memory tracking, diagnostic VM options, Arrow allocator debugging, and configurable DataFusion memory pool limits, allowing for fine-tuned performance and resource management.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/datafusion-e2e-test.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant amount of new code for Data Fusion integration, including new modules for Parquet data format and a DataFusion engine plugin. The changes span across Java, Gradle, and Rust. While this is a substantial feature addition, the review has identified several critical and high-severity issues that must be addressed. These include a security vulnerability related to arbitrary code execution in a Gradle script, incorrect logic in core components, disabled API validation checks, and the use of temporary debugging flags that disable assertions. There are also a number of medium-severity issues concerning code quality, maintainability, and potential bugs. The pull request appears to be in an early stage of development and requires thorough review and fixes before it can be considered for merging.

[gemini-code-assist]

The JVM argument -da disables assertions. The accompanying TODO comment suggests this is a temporary debugging measure. Merging this change would disable important safety checks in tests, potentially hiding bugs. This line must be removed before this pull request is merged.

[gemini-code-assist]

This ParquetDataSourceCodec incorrectly returns DataFormat.CSV. It should return DataFormat.PARQUET.

Suggested change

	return DataFormat.CSV;
	return DataFormat.PARQUET;

[gemini-code-assist]

The equals() method always returns true, and hashCode() always returns 0. This violates the Java object contract and can lead to incorrect behavior in collections like HashMap or HashSet. Since this class appears to be stateless, it could be implemented as a singleton, or equals() should perform a proper type check.

Suggested change

	public boolean equals(Object obj) {
	return true;
	}
	@Override
	public int hashCode() {
	return 0;
	}
	public boolean equals(Object obj) {
	return obj instanceof ParquetDataFormat;
	}
	@Override
	public int hashCode() {
	return getClass().hashCode();
	}

[gemini-code-assist]

The use of Eval.me(remotePlugins) introduces a critical security vulnerability. It allows for arbitrary code execution, as a malicious user could provide a crafted string to the remotePlugins property. This should be replaced with a safer parsing mechanism, such as splitting the string by a delimiter.

      def remotePluginsList = remotePlugins.split(',').collect { it.trim() }

[gemini-code-assist]

The core logic of the process method has been commented out, which effectively disables the ApiAnnotationProcessor. This processor is crucial for validating API annotations like @PublicApi and @ExperimentalApi. Disabling it compromises API contract enforcement and should be reverted.

[gemini-code-assist]

This comment states that MutexFileMetadataCache has been removed, but the struct is defined immediately after. The comment is outdated and should be removed or updated to avoid confusion.

[gemini-code-assist]

Using println! for logging is not ideal. It's better to use the log crate, which is already a dependency. This allows for structured logging with configurable levels and can be routed to the Java logging framework.

Suggested change

	Arc::new(RuntimeManager::new(cpu_threads as usize))
	info!("Runtime manager initialized with {} CPU threads", cpu_threads);

[gemini-code-assist]

Using println! for error logging is not ideal. It's better to use the log crate (e.g., log::error!) for structured logging, which will provide better visibility and control over log output.

Suggested change

	println!("[CACHE ERROR] Failed to cache metadata for: {}", file_path);
	log::error!("[CACHE ERROR] Failed to cache metadata for: {}", file_path);

[gemini-code-assist]

The use of streams to collect settings here is unnecessarily complex. This can be simplified by directly adding the setting lists to settingList, which would improve readability.

Suggested change

	return settingList;
	}
	settingList.addAll(CacheSettings.CACHE_SETTINGS);
	settingList.addAll(CacheSettings.CACHE_ENABLED);

[gemini-code-assist]

This line is commented out. It appears to be intended for dynamically updating cache size limits. It should either be implemented to enable this functionality or removed to keep the code clean.