LLM-Value-Eval-Framework

A robust, modular automated evaluation framework for LLM value alignment research.

📖 中文文档 (Chinese)

Introduction

LLM-Value-Eval-Framework is a production-grade evaluation framework designed for researchers and engineers studying value alignment in Large Language Models. It provides a complete pipeline from multi-model API orchestration to advanced semantic analysis, enabling systematic evaluation of how different LLMs respond to value-laden scenarios.

Key Capabilities

• 🚀 High-Concurrency Infrastructure: Execute thousands of API calls efficiently with hierarchical concurrency control
• 🔄 Resilient Execution: Smart API key rotation, auto-retry mechanisms, and checkpoint-based caching
• 📊 Advanced Analysis: Multi-dimensional similarity computation and clustering for value alignment research
• 🏗️ Modular Architecture: Clean separation of concerns following SOLID principles

Architecture

graph TB
    subgraph "User Layer"
        A[main.py] --> B[config.yaml]
        A --> C[prompts/*.json]
    end
    
    subgraph "Orchestration Layer"
        D[ExperimentRunner]
        D --> E[Task Generation]
        D --> F[Result Collection]
    end
    
    subgraph "Execution Layer"
        G[ParallelExecutor]
        H[CacheManager]
        G --> I[ThreadPoolExecutor]
        H --> J[Checkpoint & Resume]
    end
    
    subgraph "Resource Control Layer"
        K[ProviderConcurrencyManager]
        L[RateLimiter]
        K --> M[Provider Semaphores]
        L --> N[Token Bucket]
    end
    
    subgraph "API Layer"
        O[APIClient]
        O --> P[Key Rotation]
        O --> Q[Error Handling]
        O --> R[Stream Processing]
    end
    
    subgraph "Analysis Layer"
        S[ValueAnalyzer]
        S --> T[SimilarityAnalyzer]
        S --> U[ClusteringAnalyzer]
        S --> V[VisualizationEngine]
    end
    
    A --> D
    D --> G
    D --> H
    G --> K
    G --> L
    K --> O
    L --> O
    O --> W[LLM APIs]
    
    F --> S

Loading

Concurrency Control Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Global ThreadPool                         │
│                   (CPU-Adaptive Workers)                     │
│  ┌─────────────────────────────────────────────────────┐    │
│  │                                                      │    │
│  │   ┌──────────┐  ┌──────────┐  ┌──────────┐         │    │
│  │   │ Provider │  │ Provider │  │ Provider │   ...   │    │
│  │   │Semaphore │  │Semaphore │  │Semaphore │         │    │
│  │   │  (n=3)   │  │  (n=10)  │  │  (n=5)   │         │    │
│  │   └────┬─────┘  └────┬─────┘  └────┬─────┘         │    │
│  │        │             │             │                │    │
│  │   ┌────▼─────┐  ┌────▼─────┐  ┌────▼─────┐         │    │
│  │   │  Token   │  │  Token   │  │  Token   │         │    │
│  │   │  Bucket  │  │  Bucket  │  │  Bucket  │         │    │
│  │   │(RPM=30)  │  │(RPM=60)  │  │(RPM=100) │         │    │
│  │   └────┬─────┘  └────┬─────┘  └────┬─────┘         │    │
│  │        │             │             │                │    │
│  │        ▼             ▼             ▼                │    │
│  │   ┌─────────────────────────────────────────┐      │    │
│  │   │           API Client Pool               │      │    │
│  │   │    (Key Rotation + Error Recovery)      │      │    │
│  │   └─────────────────────────────────────────┘      │    │
│  │                                                      │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

Key Features

🚀 Production-Grade Concurrency

• Hierarchical Control: Global ThreadPool + Provider-level Semaphores
• Dynamic Scaling: CPU-adaptive worker count (aggressive/standard/conservative modes)
• Token Bucket Rate Limiting: Per-model RPM enforcement with thread-safe implementation

# Automatic concurrency management
with concurrency_manager.acquire_provider_slot("OpenAI"):
    response = rate_limited_call(model, prompt, params)

🔄 Resilience & Fault Tolerance

• Smart Key Rotation: Circular queue with O(1) rotation and failure tracking
• Auto-Retry: Handles 20+ error types including rate limits, auth failures, and quota exceeded
• Checkpoint & Resume: Configuration-hash based caching with incremental updates

# Automatic resume from interruption
$ python main.py
🔄 Found cached experiment data:
   📁 Config hash: a1b2c3d4e5f6
   ✅ Completed tasks: 150
   💾 Resuming from cache...

📊 Advanced Analysis

• Multi-dimensional Similarity: SBERT embeddings + optional ERNIE/Tencent word vectors
• Clustering Algorithms: DBSCAN for density-based clustering, Hierarchical for interpretability
• Rich Visualizations: t-SNE plots, similarity heatmaps, force-directed graphs

🏗️ Modular Architecture

Module	Responsibility
core/experiment.py	Experiment orchestration and task management
core/api_client.py	API calls with key rotation and error handling
core/parallel_executor.py	Concurrent execution with progress monitoring
core/cache_manager.py	Checkpoint-based caching system
core/rate_limiter.py	Token bucket rate limiting
analysis/similarity.py	Multi-dimensional similarity computation
analysis/clustering.py	DBSCAN and hierarchical clustering
analysis/visualization.py	Heatmaps, t-SNE, and network graphs

Quick Start

Installation

# Clone the repository
git clone https://github.com/yourusername/LLM-Value-Eval-Framework.git
cd LLM-Value-Eval-Framework

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Configuration

# Copy the example configuration
cp config.yaml.example config.yaml

# Edit config.yaml and add your API keys

Example configuration:

experiment:
  rounds: 3
  languages: ["zh", "en"]
  output_dir: "results"

global:
  default_rpm: 60

apis:
  - provider: "OpenAI"
    base_url: "https://api.openai.com/v1"
    api_keys:
      - "sk-your-api-key-here"
    models:
      - api_name: "gpt-4o"
        alias: "GPT-4o"
        param_combos:
          - {temperature: 0.7, top_p: 0.9}
        rpm: 60

Running Experiments

# Run the experiment
python main.py

# Check cache statistics
python main.py --cache-stats

# Clean cache and restart
python main.py --clean-cache

Using the Analysis Module

from sentence_transformers import SentenceTransformer
from analysis import ValueAnalyzer

# Initialize analyzer
sbert_model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')
analyzer = ValueAnalyzer(sbert_model)

# Load experiment results
analyzer.load_experiment_results("results/output_zh.xlsx")

# Load value vocabulary data
# Replace with your own data file, or use the example: data/example_values.xlsx
analyzer.load_value_data("data/values.xlsx", "AIGC_Values", "aigc")
analyzer.load_value_data("data/values.xlsx", "Literature_Values", "literature")

# Run analysis
report = analyzer.analyze_all(
    internal_data_key="aigc",
    cross_data_keys=("literature", "aigc"),
    threshold=0.7,
    generate_viz=True
)

print(report)

Tech Stack

Category	Technologies
Language	Python 3.8+
API Integration	OpenAI SDK, OpenAI-compatible APIs
Concurrency	ThreadPoolExecutor, Semaphore, Threading Locks
NLP/ML	Sentence-Transformers, scikit-learn, PyTorch
Data Processing	Pandas, NumPy
Visualization	Matplotlib, Seaborn, NetworkX
Caching	JSON-based with MD5 hash invalidation

Supported API Providers

The framework supports any OpenAI-compatible API, including:

• OpenAI (GPT-4, GPT-4o, o1, o3)
• Anthropic Claude (via proxy)
• Google Gemini (via proxy)
• DeepSeek
• Alibaba Qwen
• ByteDance Doubao
• Tencent Hunyuan
• iFlytek Spark
• Moonshot Kimi
• Baidu ERNIE
• Meta Llama (via OpenRouter)

💡 Support for DeepSeek, Qwen, Doubao, and other Chinese LLMs is provided via OpenAI-compatible endpoint configuration in config.yaml. Simply set the appropriate base_url and api_keys for each provider.

Project Structure

LLM-Value-Eval-Framework/
├── README.md                 # This file
├── requirements.txt          # Python dependencies
├── config.yaml.example       # Configuration template
├── main.py                   # Main entry point
├── core/                     # Core infrastructure
│   ├── __init__.py
│   ├── config_loader.py      # YAML configuration loading
│   ├── experiment.py         # Experiment orchestration
│   ├── api_client.py         # API client with key rotation
│   ├── parallel_executor.py  # Concurrent task execution
│   ├── cache_manager.py      # Checkpoint & resume system
│   ├── rate_limiter.py       # Token bucket rate limiting
│   ├── provider_concurrency_manager.py  # Provider semaphores
│   ├── prompt_loader.py      # Prompt template loading
│   └── result_writer.py      # Excel output formatting
├── analysis/                 # Analysis module
│   ├── __init__.py
│   ├── similarity.py         # Multi-dimensional similarity
│   ├── clustering.py         # DBSCAN & hierarchical clustering
│   ├── visualization.py      # Heatmaps, t-SNE, graphs
│   └── value_analyzer.py     # High-level analysis interface
├── prompts/                  # Prompt templates
│   ├── zh_prompts.json       # Chinese prompts
│   └── en_prompts.json       # English prompts
├── data/                     # Data directory
│   └── example_values.xlsx   # Example value vocabulary
└── results/                  # Output directory (generated)
    └── .cache/               # Cache files (generated)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citation

If you use this framework in your research, please cite:

@software{llm_value_eval_framework,
  author = {Louis Lin},
  title = {LLM-Value-Eval-Framework: A Modular Evaluation Framework for LLM Value Alignment},
  year = {2025},
  url = {https://github.com/LouisUltra/LLM-Value-Eval-Framework}
}

Acknowledgments

• Sentence-Transformers for multilingual embeddings
• OpenAI for the API client design inspiration
• All the LLM providers for their APIs

---

Made with ❤️ for AI Safety Research