feat: add data_selection module with multiple filtering baselines

[jjk9090]

[PR] Introduction of Data-Selection Module

1. Overview

This PR introduces the data_selection module, a comprehensive and extensible pipeline designed for Data-Centric AI research. It standardizes the end-to-end process—from raw data preparation to filtering, training, and evaluation—facilitating the reproducibility of Large Language Model (LLM) instruction tuning experiments.

2. Key Features

Unified Preprocessing (preprocess_data.py)

• Multi-format Support: Seamless conversion of diverse formats (JSON, JSONL, CSV, and Parquet) into the standardized Alpaca format (instruction/input/output).

Integrated Filtering Baselines

Successfully integrated three state-of-the-art (SOTA) data selection methods as Git submodules:

• DataFlow-Agent (DFA)
• DataComp for LM (DCLM)
• Cherry_LLM

Automated Workflow (pipeline.py)

• End-to-End Execution: Supports full-cycle automation (filter → train → eval).
• Modular Control: Supports step-by-step debugging and execution via the --stage parameter.

Decoupled Architecture

• Traceability: Outputs are strictly organized into output/data/ (filtered datasets) and output/experiments/ (logs & checkpoints) to ensure a clean source tree.
• Environment Isolation: Provided specific configuration guidelines for each baseline to prevent dependency conflicts (e.g., separate Conda environments).

3. Supported Datasets

The following mainstream datasets have been tested and verified with the new preprocessing logic:

• ✅ LMSYS-Chat-1M
• ✅ WildChat
• ✅ OpenHermes-2.5
• ✅ Databricks-Dolly-15K
• ✅ Evol-Instruct 70K