Skip to content

README.md

Latest commit

 

History

History
274 lines (200 loc) · 7.38 KB

README.md

File metadata and controls

274 lines (200 loc) · 7.38 KB
Raw

Feature-Focused Examples

Each example demonstrates ONE specific BenchBox capability

This directory contains focused examples that each teach a single feature or capability of BenchBox. Unlike the getting_started examples which provide end-to-end workflows, these examples isolate specific features to make learning easier.

Philosophy

  • One feature per example - Each script focuses on a single capability
  • Practical and runnable - All examples are complete, working programs
  • Well-commented - Inline documentation explains why, not just what
  • Build on basics - Assumes you've completed getting_started examples

Available Examples

1. Test Types (test_types.py)

Feature: Different test execution types (power, throughput, maintenance) Learn: When to use each test type

python features/test_types.py

Key Concepts:

  • Power test: Sequential query execution for latency measurement
  • Throughput test: Concurrent streams for capacity testing
  • Maintenance test: Data modification operations

2. Maintenance Workflow (maintenance_workflow.py)

Feature: Complete workflow with Maintenance Test and database reload Learn: Proper test sequencing and database state management

python features/maintenance_workflow.py

Key Concepts:

  • Complete Power → Throughput → Maintenance sequence
  • Database state verification (row count tracking)
  • Why reload is required after Maintenance
  • Correct vs incorrect test workflows
  • Interactive demonstration of data modifications

Important: This example shows the critical reload requirement after Maintenance tests.

3. Query Subset (query_subset.py)

Feature: Run specific queries instead of full benchmark suite Learn: Targeted testing for development and debugging

python features/query_subset.py

Key Concepts:

  • --queries flag usage
  • Quick smoke tests (2-3 queries)
  • Debugging specific slow queries
  • CI/CD integration with fast subsets

3. Tuning Comparison (tuning_comparison.py)

Feature: Compare tuned vs baseline performance Learn: Quantify optimization impact

python features/tuning_comparison.py

Key Concepts:

  • Tuning configuration files
  • Before/after comparison
  • Performance improvement measurement
  • Cost/benefit analysis of optimizations

4. Result Analysis (result_analysis.py)

Feature: Load and compare benchmark results Learn: Analyze performance changes over time

python features/result_analysis.py

Key Concepts:

  • Result JSON format
  • Loading previous results
  • Query-by-query comparison
  • Regression detection
  • Statistical analysis

5. Multi-Platform (multi_platform.py)

Feature: Run same benchmark on multiple platforms Learn: Platform comparison workflow

python features/multi_platform.py

Key Concepts:

  • Platform iteration
  • Result collection across platforms
  • Performance comparison
  • Platform selection decision-making

6. Export Formats (export_formats.py)

Feature: Export results in JSON, CSV, HTML Learn: Result export and reporting

python features/export_formats.py

Key Concepts:

  • --formats flag usage
  • JSON for programmatic processing
  • CSV for spreadsheet analysis
  • HTML for stakeholder reports

7. Data Validation (data_validation.py)

Feature: Enable data quality checks Learn: Ensure data integrity

python features/data_validation.py

Key Concepts:

  • Preflight validation (before generation)
  • Postgen validation (after generation)
  • Postload validation (after database load)
  • Row count verification
  • Data quality checks

8. Performance Monitoring (performance_monitoring.py)

Feature: Monitor system resources during execution Learn: Resource usage tracking

python features/performance_monitoring.py

Key Concepts:

  • System profiling
  • CPU usage tracking
  • Memory usage tracking
  • Resource requirement estimation

Learning Path

Recommended order for learning features:

  1. Start: test_types.py - Understand different execution modes
  2. Next: query_subset.py - Learn targeted testing
  3. Then: tuning_comparison.py - See optimization impact
  4. Advanced: result_analysis.py - Analyze performance changes
  5. Multi-platform: multi_platform.py - Compare databases
  6. Reporting: export_formats.py - Generate reports
  7. Quality: data_validation.py - Ensure correctness
  8. Performance: performance_monitoring.py - Track resources

Usage Patterns

Quick Feature Test

# Run a feature example to see it in action
python features/query_subset.py

# Most examples complete in < 1 minute

Modify and Experiment

# Copy an example to experiment
cp features/tuning_comparison.py my_experiment.py

# Modify parameters, scale factors, platforms
# All examples are self-contained

Integration in Your Code

# Feature examples show patterns you can use in your code
# Copy the relevant sections into your scripts

Example Structure

All feature examples follow a consistent structure:

"""
Brief description of the feature.

Usage:
    python features/example.py

Key Concepts:
    - Concept 1
    - Concept 2
"""

# 1. Setup section with clear comments
# 2. Feature demonstration with explanations
# 3. Result display showing what to look for
# 4. Tips section with best practices

Prerequisites

  • Complete getting_started examples first
  • Understand basic BenchBox concepts
  • Python 3.10+ installed
  • BenchBox installed: uv add benchbox

Next Steps

After mastering these features, explore:

Tips

  1. Run examples as-is first - See the feature in action before modifying
  2. Read inline comments - They explain why, not just what
  3. Experiment with parameters - Change scale factors, platforms, etc.
  4. Combine features - Use multiple features together in your workflows
  5. Check exit codes - Examples return 0 on success for CI/CD integration

Common Questions

Q: Can I combine multiple features? A: Yes! See PATTERNS.md for combinations.

Q: How do these relate to unified_runner.py? A: These examples show individual features. unified_runner.py combines all features in one tool.

Q: Should I use these in production? A: These are educational examples. For production, use unified_runner.py or build your own based on these patterns.

Q: Can I modify these examples? A: Absolutely! They're designed to be copied and adapted for your needs.

Contributing

Have a feature that needs a focused example? Submit a PR following this pattern:

  1. One feature per example
  2. Complete, runnable code
  3. Inline comments explaining concepts
  4. Usage section in docstring
  5. Update this README with your example

Remember: Each example teaches ONE feature in depth. For complete workflows combining features, see PATTERNS.md.