From 7d5c041a04cbf310ee7856a732bd4190e885fee4 Mon Sep 17 00:00:00 2001 From: c7w Date: Tue, 30 Jun 2026 20:37:28 +0800 Subject: [PATCH 1/3] Add Medal Score metric, v1-lite split, and released score artifacts - Medal Score: peer-relative gold/silver/bronze podium (normalized to [0,1]), reported on v1 (47 tasks) and the v1-lite subset (10 tasks). READMEs now lead with Medal Score; average rank stays on the website leaderboard. - leaderboard/: ship the frozen podium baselines (medal_podium.csv), published leaderboard (medal_leaderboard.csv), raw score table (exp1_models_raw.csv), a submission scorer (score_submission.py), and an example submission. Un-ignore leaderboard/*.csv. - v1-lite: add frontier_eval/conf/batch/v1_lite.yaml (10-task subset across all five categories, distinct families, gradual-improvement tasks). Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 2 + README.md | 38 +++++--- README_zh-CN.md | 31 ++++--- frontier_eval/conf/batch/v1_lite.yaml | 83 +++++++++++++++++ leaderboard/README.md | 62 +++++++++++++ leaderboard/exp1_models_raw.csv | 48 ++++++++++ leaderboard/medal_leaderboard.csv | 9 ++ leaderboard/medal_podium.csv | 48 ++++++++++ leaderboard/score_submission.py | 124 ++++++++++++++++++++++++++ leaderboard/submission_example.csv | 48 ++++++++++ 10 files changed, 469 insertions(+), 24 deletions(-) create mode 100644 frontier_eval/conf/batch/v1_lite.yaml create mode 100644 leaderboard/README.md create mode 100644 leaderboard/exp1_models_raw.csv create mode 100644 leaderboard/medal_leaderboard.csv create mode 100644 leaderboard/medal_podium.csv create mode 100644 leaderboard/score_submission.py create mode 100644 leaderboard/submission_example.csv diff --git a/.gitignore b/.gitignore index 8a45ae5c..bf3b45cf 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ benchmarks/Astrodynamics/MannedLunarLanding/results.txt benchmarks/Astrodynamics/MannedLunarLanding/outputlog.txt .vscode *.csv +!leaderboard/*.csv *.h5ad *.h5ad.tmp *.h5ad.tmp.* @@ -46,6 +47,7 @@ outputlog.txt frontier_eval/conf/batch/* !frontier_eval/conf/batch/example_matrix.yaml !frontier_eval/conf/batch/v1.yaml +!frontier_eval/conf/batch/v1_lite.yaml metrics.json artifacts.json debug-*.log diff --git a/README.md b/README.md index 0fdf60bf..98acef96 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,10 @@ Frontier-Eng is a benchmark for **generative optimization**: agents iteratively The benchmark currently covers **47 tasks** across computing, quantum information, operations research, robotics and control, optics and communications, and physical sciences. The project homepage and paper frame it as a missing evaluation axis between pass/fail coding benchmarks and real engineering work: most engineering problems start from a feasible baseline and reward iterative improvement, not one-shot correctness. +## News + +- **2026-06-30** — **New scoring metric: the Medal Score (gold/silver/bronze).** Alongside average rank, we now release a peer-relative *Medal Score* (normalized to `[0,1]`). On each task the top-3 best-feasible scores in the v1 snapshot are frozen as gold/silver/bronze baselines; a model earns 1.00 / 0.67 / 0.33 for reaching each, averaged over the task set, and is reported on both v1 (47 tasks) and v1-lite (10 tasks). It rewards only reaching each task's frontier and ignores negligible long-tail margins, making cross-task aggregation fairer. Per-task podium values and the leaderboard live in [`leaderboard/`](leaderboard/README.md). +- **2026-06-30** — **`v1-lite` released.** A 10-task representative subset of `v1` covering all five categories with distinct benchmark families, selected for tasks whose scores climb gradually under budget (not one-shot-saturated or all-or-nothing). Run it with `frontier_eval/conf/batch/v1_lite.yaml`. ## 0. Host Requirements @@ -97,6 +101,14 @@ bash scripts/batch/validate_v1_task_envs.sh That command runs the batch config for the `v1` problem set with `algorithm.iterations=0`, which evaluates each task's shipped baseline instead of asking an LLM to improve it. +### `v1-lite` quick subset + +For fast iteration and ablations, use the 10-task `v1-lite` matrix +([`frontier_eval/conf/batch/v1_lite.yaml`](frontier_eval/conf/batch/v1_lite.yaml)) +instead of the full `v1` config. It spans all five categories with distinct +benchmark families and favors tasks whose scores improve gradually under budget, +so a short run still exercises the full optimization loop. + If you want the full `v1` problem set with normal optimization runs later, see [`run.md`](run.md). ## Where To Go Next @@ -108,18 +120,20 @@ If you want the full `v1` problem set with normal optimization runs later, see [ ## Leaderboard -Detailed leaderboard: [lab.einsia.ai/frontier-eng/leaderboard.html](https://lab.einsia.ai/frontier-eng/leaderboard.html) - -| Rank | Model | Average Rank | -| :--: | :--- | --: | -| 1 | GPT-5.4 | 3.54 | -| 2 | Claude Opus 4.6 | 3.63 | -| 3 | GLM-5 | 4.34 | -| 4 | DeepSeek V3.2 | 4.76 | -| 5 | Gemini 3.1 Pro Preview | 5.53 | -| 6 | Grok 4.20 | 5.82 | -| 7 | SEED 2.0 Pro | 5.86 | -| 8 | Qwen3 Coder Next | 6.71 | +Detailed leaderboard (incl. average rank): [lab.einsia.ai/frontier-eng/leaderboard](https://lab.einsia.ai/frontier-eng/leaderboard). Released score tables and the per-task medal podium: [`leaderboard/`](leaderboard/README.md). + +**Medal Score** (gold/silver/bronze podium, higher is better, normalized to `[0,1]` = mean per-task podium credit). On each task the top-3 best scores in the **v1 snapshot (2026-04-14)** are frozen as gold/silver/bronze baselines; a model earns 1.00 / 0.67 / 0.33 for reaching each. Reported on both the full **v1** set (47 tasks) and the **v1-lite** subset (10 tasks); gold/silver/bronze counts are for v1 (see [`leaderboard/`](leaderboard/README.md)): + +| Rank | Model | Medal (v1) | Medal (v1-lite) | 🥇 | 🥈 | 🥉 | +| :--: | :--- | --: | --: | --: | --: | --: | +| 1 | GPT-5.4 | 0.596 | 0.667 | 24 | 5 | 2 | +| 2 | Claude Opus 4.6 | 0.490 | 0.501 | 9 | 18 | 6 | +| 3 | GLM-5 | 0.312 | 0.233 | 4 | 10 | 12 | +| 4 | DeepSeek V3.2 | 0.248 | 0.166 | 3 | 9 | 8 | +| 5 | Gemini 3.1 Pro Preview | 0.213 | 0.200 | 3 | 6 | 9 | +| 6 | Seed 2.0 Pro | 0.185 | 0.100 | 3 | 7 | 3 | +| 7 | Grok 4.20 | 0.184 | 0.133 | 3 | 6 | 5 | +| 8 | Qwen3 Coder Next | 0.121 | 0.000 | 3 | 3 | 2 | ## Contributing diff --git a/README_zh-CN.md b/README_zh-CN.md index 75ddd213..26e66e5b 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -8,6 +8,11 @@ Frontier-Eng 是一个面向 **generative optimization** 的 benchmark：Agent 当前版本包含 **47 个任务**，覆盖计算系统、量子信息、运筹优化、机器人控制、光学通信、物理与工程设计。主页和论文的核心观点是：真实工程问题通常从一个可行 baseline 出发，价值来自持续优化，而不是 pass/fail。 +## News + +- **2026-06-30** — **新增评测指标：金银铜 Medal Score。** 在 average rank 之外，我们发布同侪相对的 *Medal Score*（归一化到 `[0,1]`）：每道题取 v1 snapshot 中最好的前三名分数冻结为金/银/铜 baseline，模型达到金/银/铜分别得 1.00 / 0.67 / 0.33，对题集求均值；同时汇报 v1（47 题）与 v1-lite（10 题）。它只奖励"达到该题最前沿（领奖台）"，忽略长尾里可忽略的微小差距，使跨题汇总更公平。每题 podium 分数与榜单见 [`leaderboard/`](leaderboard/README.md)。 +- **2026-06-30** — **发布 `v1-lite`。** `v1` 的 10 题代表性子集，覆盖全部五大类、family 各不相同，专选"分数随预算逐步提升（而非一步做满或非高即低）"的题，配置见 `frontier_eval/conf/batch/v1_lite.yaml`。 + ## 这个 benchmark 在测什么 和传统 agent benchmark 相比，Frontier-Eng 更关注三件事： @@ -110,18 +115,20 @@ bash scripts/batch/validate_v1_task_envs.sh ## Leaderboard -详细榜单见 [lab.einsia.ai/frontier-eng/leaderboard.html](https://lab.einsia.ai/frontier-eng/leaderboard.html)。 - -| 排名 | Model | Average Rank | -| :--: | :--- | --: | -| 1 | Claude Opus 4.6 | 3.18 | -| 2 | GLM-5 | 4.02 | -| 3 | DeepSeek V3.2 | 4.41 | -| 4 | Gemini 3.1 Pro Preview | 5.34 | -| 5 | Grok 4.20 | 5.60 | -| 6 | SEED 2.0 Pro | 5.63 | -| 7 | GPT-5.4 | 5.68 | -| 8 | Qwen3 Coder Next | 6.68 | +详细榜单（含 average rank）见 [lab.einsia.ai/frontier-eng/leaderboard](https://lab.einsia.ai/frontier-eng/leaderboard)。发布的分数表与每题金银铜 podium 见 [`leaderboard/`](leaderboard/README.md)。 + +**Medal Score**（金银铜 podium，越高越好，归一化到 `[0,1]`，即每题领奖台得分的均值）。每题取 **v1 snapshot (2026-04-14)** 的前三名分数冻结为金/银/铜 baseline，模型达到金/银/铜分别得 1.00 / 0.67 / 0.33。同时汇报 **v1**（47 题）与 **v1-lite**（10 题）两个集合；金银铜次数为 v1（`gpt-5.4` 采用其 47 题全量重测结果）： + +| 排名 | Model | Medal (v1) | Medal (v1-lite) | 🥇 | 🥈 | 🥉 | +| :--: | :--- | --: | --: | --: | --: | --: | +| 1 | GPT-5.4 | 0.596 | 0.667 | 24 | 5 | 2 | +| 2 | Claude Opus 4.6 | 0.490 | 0.501 | 9 | 18 | 6 | +| 3 | GLM-5 | 0.312 | 0.233 | 4 | 10 | 12 | +| 4 | DeepSeek V3.2 | 0.248 | 0.166 | 3 | 9 | 8 | +| 5 | Gemini 3.1 Pro Preview | 0.213 | 0.200 | 3 | 6 | 9 | +| 6 | Seed 2.0 Pro | 0.185 | 0.100 | 3 | 7 | 3 | +| 7 | Grok 4.20 | 0.184 | 0.133 | 3 | 6 | 5 | +| 8 | Qwen3 Coder Next | 0.121 | 0.000 | 3 | 3 | 2 | ## 贡献 diff --git a/frontier_eval/conf/batch/v1_lite.yaml b/frontier_eval/conf/batch/v1_lite.yaml new file mode 100644 index 00000000..284cc45a --- /dev/null +++ b/frontier_eval/conf/batch/v1_lite.yaml @@ -0,0 +1,83 @@ +# Frontier-Eng v1-lite batch matrix. +# +# v1-lite is a 10-task representative subset of the full 47-task v1 set, chosen +# for fast iteration and ablation. Tasks are selected to (a) cover all five +# engineering categories with distinct benchmark families and (b) reward +# incremental optimization -- their best-feasible scores climb gradually with +# budget instead of saturating in one shot or splitting all-or-nothing. The +# selection rule (top-graduality task per family, 2 per category) is documented +# and reproduced by Einsia-lab/scripts/build_frontier_metrics.py. +# +# Overrides for each task mirror frontier_eval/conf/batch/v1.yaml exactly. +# +version: 1 +tasks: +- name: unified + label: ComputerSystems/MallocLab + overrides: + - task.benchmark=ComputerSystems/MallocLab +- name: unified + label: QuantumComputing/task_01_routing_qftentangled + overrides: + - task.benchmark=QuantumComputing/task_01_routing_qftentangled + - task.runtime.env_name=frontier-v1-main +- name: unified + label: JobShop/abz + overrides: + - task.benchmark=JobShop/abz + - task.runtime.python_path=uv-env:frontier-v1-main +- name: unified + label: InventoryOptimization/disruption_eoqd + overrides: + - task.benchmark=InventoryOptimization/disruption_eoqd + - task.runtime.env_name=frontier-v1-main +- name: unified + label: EnergyStorage/BatteryFastChargingSPMe + overrides: + - task.benchmark=EnergyStorage/BatteryFastChargingSPMe +- name: unified + label: Robotics/RobotArmCycleTimeOptimization + overrides: + - task.benchmark=Robotics/RobotArmCycleTimeOptimization + - task.runtime.env_name=frontier-v1-main + - algorithm.oe.evaluator.timeout=600 +- name: unified + label: Optics/holographic_multiplane_focusing + overrides: + - task.benchmark=Optics/holographic_multiplane_focusing + - task.runtime.env_name=frontier-v1-main + - algorithm.oe.evaluator.timeout=600 +- name: unified + label: WirelessChannelSimulation/HighReliableSimulation + overrides: + - task.benchmark=WirelessChannelSimulation/HighReliableSimulation +- name: unified + label: ReactionOptimisation/snar_multiobjective + overrides: + - task.benchmark=ReactionOptimisation/snar_multiobjective + - task.runtime.python_path=uv-env:frontier-v1-summit + - algorithm.oe.evaluator.timeout=600 +- name: unified + label: StructuralOptimization/TopologyOptimization + overrides: + - task.benchmark=StructuralOptimization/TopologyOptimization + - task.runtime.env_name=frontier-v1-main +algorithms: +- name: openevolve + overrides: + - algorithm.iterations=100 + - algorithm.oe.evaluator.timeout=300 +llms: +- name: v1 + api_base: ${oc.env:OPENAI_API_BASE,https://api.openai.com/v1} + model: ${oc.env:OPENAI_MODEL,gpt-4o-mini} + api_key_env: OPENAI_API_KEY +llm_config: openai_compatible +common_overrides: +- llm.temperature=0.7 +- llm.timeout=60 +run: + name: v1_lite + base_dir: runs/batch + max_parallel: 4 + fail_fast: false diff --git a/leaderboard/README.md b/leaderboard/README.md new file mode 100644 index 00000000..25e039b9 --- /dev/null +++ b/leaderboard/README.md @@ -0,0 +1,62 @@ +# Leaderboard & Medal Score + +Released score artifacts for the Frontier-Eng `v1` set (Experiment 1: foundation +models under `openevolve`, 100 iterations, same initial programs and frozen +verifiers; `gpt-5.4` uses its full 47-task retest). + +| File | Contents | +|---|---| +| `medal_podium.csv` | Frozen per-task **gold / silver / bronze** threshold scores and the model that set each. | +| `medal_leaderboard.csv` | Per-model normalized **Medal Score** on v1 and v1-lite, with gold/silver/bronze counts. | +| `exp1_models_raw.csv` | Best-feasible score of each model on each of the 47 tasks (higher is better); source of the podium. | +| `score_submission.py` | Scores a new submission against the frozen podium. | +| `submission_example.csv` | Example submission (claude-opus-4.6) — scoring it reproduces its leaderboard line. | + +## Medal Score + +On each task the top-3 best scores in the **v1 snapshot (2026-04-14)** are frozen +as peer baselines — gold (1st), silver (2nd), bronze (3rd). A model earns +**1.00** for reaching the gold score, **0.67** for silver, **0.33** for bronze, +otherwise 0; its Medal Score is the **mean** of this credit over a task set +(normalized to `[0,1]`). It credits only reaching each task's frontier (the +podium) and ignores negligible margins in the long tail — a fairer aggregate +than crediting every ordinal rank when the question is "how often does a model +reach the best-known solutions?" We report it on both the full **v1** set +(47 tasks) and the **v1-lite** subset (10 tasks). (Average rank and other +diagnostics are on the [website leaderboard](https://lab.einsia.ai/frontier-eng/leaderboard).) + +> `gpt-oss-120b` is part of the paper's 9-model rank tables, but its per-task raw +> scores were not retained; the released podium is therefore computed over the 8 +> models with available raw scores. + +## Medal leaderboard (normalized; gold/silver/bronze counts are for v1) + +| Rank | Model | Medal (v1) | Medal (v1-lite) | 🥇 | 🥈 | 🥉 | +| :--: | :--- | --: | --: | --: | --: | --: | +| 1 | gpt-5.4 | 0.596 | 0.667 | 24 | 5 | 2 | +| 2 | claude-opus-4.6 | 0.490 | 0.501 | 9 | 18 | 6 | +| 3 | glm-5 | 0.312 | 0.233 | 4 | 10 | 12 | +| 4 | deepseek-v3.2 | 0.248 | 0.166 | 3 | 9 | 8 | +| 5 | gemini-3.1-pro-preview | 0.213 | 0.200 | 3 | 6 | 9 | +| 6 | seed-2.0-pro | 0.185 | 0.100 | 3 | 7 | 3 | +| 7 | grok-4.20 | 0.184 | 0.133 | 3 | 6 | 5 | +| 8 | qwen3-coder-next | 0.121 | 0.000 | 3 | 3 | 2 | + +## Score your own model + +Put your model's best score per task in a CSV (`Task,Score`, one row per task, +task names as in `medal_podium.csv`), then: + +```bash +python leaderboard/score_submission.py your_scores.csv +# -> Medal Score (v1, 47 tasks) : 0.xxx (gold .., silver .., bronze ..) +# Medal Score (v1-lite, 10 tasks) : 0.xxx +``` + +Sanity check (reproduces claude-opus-4.6's line, 0.490 / 0.501): + +```bash +python leaderboard/score_submission.py leaderboard/submission_example.csv +``` + +Interactive view: [lab.einsia.ai/frontier-eng/leaderboard](https://lab.einsia.ai/frontier-eng/leaderboard) diff --git a/leaderboard/exp1_models_raw.csv b/leaderboard/exp1_models_raw.csv new file mode 100644 index 00000000..1b720953 --- /dev/null +++ b/leaderboard/exp1_models_raw.csv @@ -0,0 +1,48 @@ +Task,Baseline,claude-opus-4.6_best,deepseek-v3.2_best,gemini-3.1-pro-preview_best,glm-5_best,gpt-5.4_best,grok-4.20_best,qwen3-coder-next_best,seed-2.0-pro_best,,,,,,,,,, +Aerodynamics_CarAerodynamicsSensing,0.9617,0.9624,0.9632,0.9632,0.9628,0.9630695838481188,0.9624,0.9632,0.9624,,,,,,,,,, +Astrodynamics_MannedLunarLanding,4577.437,6027.3126,6079.2455,4674.9462,6839.0331,6660.942428,4577.437,4577.437,4733.0435,,,,,,,,,, +ComputerSystems_MallocLab,28,96,53,48,86,28,57,32,38,,,,,,,,,, +Cryptographic_AES-128,7.5209,11.8617,12.4591,10.2396,7.9669,39.824967043300866,10.8615,5.5501,7.9481,,,,,,,,,, +Cryptographic_SHA-256,9.8274,16.7955,9.718,9.942,15.1655,26.34045367870492,17.2504,9.8475,15.2838,,,,,,,,,, +Cryptographic_SHA3-256,16.0932,17.4003,17.0749,16.2255,17.5778,37.44512785396786,16.0594,16.5292,18.3478,,,,,,,,,, +EnergyStorage_BatteryFastChargingProfile,71.2806,120.8025,111.4518,116.6532,118.7678,121.99136502281442,99.6875,89.8416,115.6882,,,,,,,,,, +EnergyStorage_BatteryFastChargingSPMe,66.1636,71.8225,91.0079,92.3198,78.0896,122.94304361063023,76.4657,79.0273,76.4122,,,,,,,,,, +EngDesign,1.3571,1.3571,21.7143,27,25.5714,1.3571428571428572,27,25.5714,27,,,,,,,,,, +InventoryOptimization_disruption_eoqd,0.3642,0.6473,0.6381,0.639,0.6303,1,0.6359,0.6225,0.6321,,,,,,,,,, +InventoryOptimization_finite_horizon_dp,0.3673,0.9596,0.8025,0.7559,0.7965,0.9606835281410351,0.8547,0.4413,0.7323,,,,,,,,,, +InventoryOptimization_general_meio,0.1825,0.9929,0.9893,0.9839,0.9165,0.9999999999999999,0.9236,0.7819,0.6973,,,,,,,,,, +InventoryOptimization_joint_replenishment,0.3034,0.8822,0.8822,0.8822,0.8822,1,0.8822,0.8821,0.8822,,,,,,,,,, +InventoryOptimization_tree_gsm_safety_stock,0.3813,0.75,0.6606,0.6606,0.6606,1,0.6606,0.6606,0.6606,,,,,,,,,, +JobShop_abz,80.5042,96.1035,88.3614,86.751,88.4924,91.23143065488635,87.6717,85.603,86.672,,,,,,,,,, +JobShop_swv,81.6325,89.4966,82.3575,82.3141,87.1611,87.33430826602005,85.5068,82.6129,82.4153,,,,,,,,,, +JobShop_ta,78.8,90.8322,84.9043,85.7065,86.8095,86.16070055174835,84.9136,85.5489,83.9694,,,,,,,,,, +KernelEngineering_FlashAttention,55.2957,983.5001,987.2034,991.8896,381.6257,182687.44188255747,324.919,525.5567,1218.5163,,,,,,,,,, +KernelEngineering_MLA,0.7828,1000.3859,0.8936,1253.2017,20.1972,1132.0659025372765,19.8651,0.9271,19.987,,,,,,,,,, +KernelEngineering_TriMul,47.1274,357.1636,85.5923,54.5774,110.8785,47.88292233116043,165.0294,49.1232,84.9069,,,,,,,,,, +Optics_adaptive_fault_tolerant_fusion,0.3959,0.6398,0.64,0.6398,0.6398,0.455046169,0.6398,0.6398,0.6398,,,,,,,,,, +Optics_adaptive_temporal_smooth_control,0.3152,0.8419,0.8419,0.8419,0.8417,0.841880414,0.842,0.8421,0.8421,,,,,,,,,, +Optics_fiber_guardband_spectrum_packing,0.3861,0.6692,0.657,0.6629,0.6692,0.6754289215686274,0.6629,0.657,0.657,,,,,,,,,, +Optics_fiber_mcs_power_scheduling,0.3297,0.6542,0.5182,0.4796,0.6491,0.6608370951757289,0.4557,0.4458,0.6491,,,,,,,,,, +Optics_fiber_wdm_channel_power_allocation,0.3255,0.6675,0.6679,0.6619,0.6686,0.6964207451370852,0.6664,0.6666,0.6654,,,,,,,,,, +Optics_holographic_multifocus_power_ratio,0.3927,0.8072,0.8265,0.5368,0.711,0.9999999999663148,0.4058,0.5875,0.5626,,,,,,,,,, +Optics_holographic_multiplane_focusing,0.3302,0.6002,0.7196,0.4398,0.4516,0.9999999999886867,0.474,0.5631,0.5303,,,,,,,,,, +Optics_phase_dammann_uniform_orders,26.8969,99.7995,97.3436,97.9498,97.8709,99.99999999999999,94.4055,95.9998,69.0576,,,,,,,,,, +Optics_phase_fourier_pattern_holography,32.6457,82.1276,74.5838,76.6371,76.0127,99.99998936790779,74.217,67.3393,72.4578,,,,,,,,,, +PyPortfolioOpt_robust_mvo_rebalance,32.9804,99.9946,84.941,77.165,82.8015,99.99460428985267,99.983,85.5194,83.0681,,,,,,,,,, +QuantumComputing_task_01_routing_qftentangled,0.209,5.0479,3.6155,0.209,3.7681,6.507945106686525,3.7655,3.2471,3.6783,,,,,,,,,, +QuantumComputing_task_02_clifford_t_synthesis,1.7134,1.6633,1.7134,1.7134,7.4236,1.7133669376223557,1.6633,1.7134,1.7134,,,,,,,,,, +QuantumComputing_task_03_cross_target_qaoa,2.4149,2.5781,5.103,2.9782,5.0301,2.4149139615375192,2.6363,2.4517,2.9782,,,,,,,,,, +ReactionOptimisation_mit_case1_mixed,87.3082,98.6621,98.6041,96.5437,95.9314,98.66214557690091,87.3082,95.3732,95.4297,,,,,,,,,, +ReactionOptimisation_reizman_suzuki_pareto,63.5202,82.3427,82.0329,79.473,82.9901,82.24612252072882,63.5202,81.4666,79.7011,,,,,,,,,, +ReactionOptimisation_snar_multiobjective,57.5234,87.3657,82.7881,80.1521,81.7614,100,72.3909,72.8477,79.427,,,,,,,,,, +Robotics_DynamicObstacleAvoidanceNavigation,0.0722,0.086,0.0856,0.0834,0.0857,0.08571428571428559,0.0817,0.0765,0.0855,,,,,,,,,, +Robotics_PIDTuning,0.0366,0.1632,0.151,0.1521,0.1515,0.1511172761100511,0.1585,0.1422,0.1514,,,,,,,,,, +Robotics_QuadrupedGaitOptimization,0.0218,0.0219,0.0749,0.0218,0.1085,0.022154337029969478,0.0227,0.0232,0.0218,,,,,,,,,, +Robotics_RobotArmCycleTimeOptimization,0.2922,0.4158,0.3923,0.4305,0.4219,0.4356212836221511,0.3923,0.3155,0.3256,,,,,,,,,, +Robotics_UAVInspectionCoverageWithWind,28.8519,28.8519,38.8024,28.8519,35.1121,30.121714802877325,55.9109,32.8468,32.1552,,,,,,,,,, +SingleCellAnalysis_predict_modality,0.5467,0.5467,0.5467,0.5467,0.5467,1,0.5467,0.5467,0.5467,,,,,,,,,, +StructuralOptimization_ISCSO2015,-5401.589,-968.4567,-1120.212,-5401.589,-1139.3354,-5401.589002,-1318.7566,-1308.2575,-1302.2288,,,,,,,,,, +StructuralOptimization_ISCSO2023,-77813242.9,-16477799.48,-55182772.3,-20092179.33,-17840974.17,-77813242.9,-30028112.28,-66126744.97,-42625693.78,,,,,,,,,, +StructuralOptimization_TopologyOptimization,-195.9153,-190.1498,-190.3706,-189.3039,-188.4673,-195.9152621,-185.7983,-192.8488,-190.0603,,,,,,,,,, +SustainableDataCenterControl_hand_written_control,8.3294,21.5657,15.292,12.9088,19.5978,8.5903,14.2432,30.1873,29.2868,,,,,,,,,, +WirelessChannelSimulation_HighReliableSimulation,192.5193,292.3228,291.9451,232.9071,248.0119,231.22403446412542,245.7082,259.9776,304.0437,,,,,,,,,, \ No newline at end of file diff --git a/leaderboard/medal_leaderboard.csv b/leaderboard/medal_leaderboard.csv new file mode 100644 index 00000000..20b35004 --- /dev/null +++ b/leaderboard/medal_leaderboard.csv @@ -0,0 +1,9 @@ +Rank,Model,Medal_v1,Medal_v1lite,Gold,Silver,Bronze +1,gpt-5.4,0.596,0.667,24,5,2 +2,claude-opus-4.6,0.49,0.501,9,18,6 +3,glm-5,0.312,0.233,4,10,12 +4,deepseek-v3.2,0.248,0.166,3,9,8 +5,gemini-3.1-pro-preview,0.213,0.2,3,6,9 +6,seed-2.0-pro,0.185,0.1,3,7,3 +7,grok-4.20,0.184,0.133,3,6,5 +8,qwen3-coder-next,0.121,0.0,3,3,2 diff --git a/leaderboard/medal_podium.csv b/leaderboard/medal_podium.csv new file mode 100644 index 00000000..8059fced --- /dev/null +++ b/leaderboard/medal_podium.csv @@ -0,0 +1,48 @@ +Task,Baseline,Gold,Gold_model,Silver,Silver_model,Bronze,Bronze_model +Aerodynamics_CarAerodynamicsSensing,0.9617,0.9632,deepseek-v3.2/gemini-3.1-pro-preview/qwen3-coder-next,0.9632,deepseek-v3.2/gemini-3.1-pro-preview/qwen3-coder-next,0.9632,deepseek-v3.2/gemini-3.1-pro-preview/qwen3-coder-next +Astrodynamics_MannedLunarLanding,4577.437,6839.0331,glm-5,6660.942428,gpt-5.4,6079.2455,deepseek-v3.2 +ComputerSystems_MallocLab,28,96.0,claude-opus-4.6,86.0,glm-5,57.0,grok-4.20 +Cryptographic_AES-128,7.5209,39.824967043300866,gpt-5.4,12.4591,deepseek-v3.2,11.8617,claude-opus-4.6 +Cryptographic_SHA-256,9.8274,26.34045367870492,gpt-5.4,17.2504,grok-4.20,16.7955,claude-opus-4.6 +Cryptographic_SHA3-256,16.0932,37.44512785396786,gpt-5.4,18.3478,seed-2.0-pro,17.5778,glm-5 +EnergyStorage_BatteryFastChargingProfile,71.2806,121.99136502281442,gpt-5.4,120.8025,claude-opus-4.6,118.7678,glm-5 +EnergyStorage_BatteryFastChargingSPMe,66.1636,122.94304361063023,gpt-5.4,92.3198,gemini-3.1-pro-preview,91.0079,deepseek-v3.2 +EngDesign,1.3571,27.0,gemini-3.1-pro-preview/grok-4.20/seed-2.0-pro,27.0,gemini-3.1-pro-preview/grok-4.20/seed-2.0-pro,27.0,gemini-3.1-pro-preview/grok-4.20/seed-2.0-pro +InventoryOptimization_disruption_eoqd,0.3642,1.0,gpt-5.4,0.6473,claude-opus-4.6,0.639,gemini-3.1-pro-preview +InventoryOptimization_finite_horizon_dp,0.3673,0.9606835281410351,gpt-5.4,0.9596,claude-opus-4.6,0.8547,grok-4.20 +InventoryOptimization_general_meio,0.1825,0.9999999999999999,gpt-5.4,0.9929,claude-opus-4.6,0.9893,deepseek-v3.2 +InventoryOptimization_joint_replenishment,0.3034,1.0,gpt-5.4,0.8822,claude-opus-4.6/deepseek-v3.2/gemini-3.1-pro-preview/glm-5/grok-4.20/seed-2.0-pro,0.8822,claude-opus-4.6/deepseek-v3.2/gemini-3.1-pro-preview/glm-5/grok-4.20/seed-2.0-pro +InventoryOptimization_tree_gsm_safety_stock,0.3813,1.0,gpt-5.4,0.75,claude-opus-4.6,0.6606,deepseek-v3.2/gemini-3.1-pro-preview/glm-5/grok-4.20/qwen3-coder-next/seed-2.0-pro +JobShop_abz,80.5042,96.1035,claude-opus-4.6,91.23143065488635,gpt-5.4,88.4924,glm-5 +JobShop_swv,81.6325,89.4966,claude-opus-4.6,87.33430826602005,gpt-5.4,87.1611,glm-5 +JobShop_ta,78.8,90.8322,claude-opus-4.6,86.8095,glm-5,86.16070055174835,gpt-5.4 +KernelEngineering_FlashAttention,55.2957,182687.44188255747,gpt-5.4,1218.5163,seed-2.0-pro,991.8896,gemini-3.1-pro-preview +KernelEngineering_MLA,0.7828,1253.2017,gemini-3.1-pro-preview,1132.0659025372765,gpt-5.4,1000.3859,claude-opus-4.6 +KernelEngineering_TriMul,47.1274,357.1636,claude-opus-4.6,165.0294,grok-4.20,110.8785,glm-5 +Optics_adaptive_fault_tolerant_fusion,0.3959,0.64,deepseek-v3.2,0.6398,claude-opus-4.6/gemini-3.1-pro-preview/glm-5/grok-4.20/qwen3-coder-next/seed-2.0-pro,0.6398,claude-opus-4.6/gemini-3.1-pro-preview/glm-5/grok-4.20/qwen3-coder-next/seed-2.0-pro +Optics_adaptive_temporal_smooth_control,0.3152,0.8421,qwen3-coder-next/seed-2.0-pro,0.8421,qwen3-coder-next/seed-2.0-pro,0.842,grok-4.20 +Optics_fiber_guardband_spectrum_packing,0.3861,0.6754289215686274,gpt-5.4,0.6692,claude-opus-4.6/glm-5,0.6692,claude-opus-4.6/glm-5 +Optics_fiber_mcs_power_scheduling,0.3297,0.6608370951757289,gpt-5.4,0.6542,claude-opus-4.6,0.6491,glm-5/seed-2.0-pro +Optics_fiber_wdm_channel_power_allocation,0.3255,0.6964207451370852,gpt-5.4,0.6686,glm-5,0.6679,deepseek-v3.2 +Optics_holographic_multifocus_power_ratio,0.3927,0.9999999999663148,gpt-5.4,0.8265,deepseek-v3.2,0.8072,claude-opus-4.6 +Optics_holographic_multiplane_focusing,0.3302,0.9999999999886867,gpt-5.4,0.7196,deepseek-v3.2,0.6002,claude-opus-4.6 +Optics_phase_dammann_uniform_orders,26.8969,99.99999999999999,gpt-5.4,99.7995,claude-opus-4.6,97.9498,gemini-3.1-pro-preview +Optics_phase_fourier_pattern_holography,32.6457,99.99998936790779,gpt-5.4,82.1276,claude-opus-4.6,76.6371,gemini-3.1-pro-preview +PyPortfolioOpt_robust_mvo_rebalance,32.9804,99.99460428985267,gpt-5.4,99.9946,claude-opus-4.6,99.983,grok-4.20 +QuantumComputing_task_01_routing_qftentangled,0.209,6.507945106686525,gpt-5.4,5.0479,claude-opus-4.6,3.7681,glm-5 +QuantumComputing_task_02_clifford_t_synthesis,1.7134,7.4236,glm-5,1.7134,deepseek-v3.2/gemini-3.1-pro-preview/qwen3-coder-next/seed-2.0-pro,1.7134,deepseek-v3.2/gemini-3.1-pro-preview/qwen3-coder-next/seed-2.0-pro +QuantumComputing_task_03_cross_target_qaoa,2.4149,5.103,deepseek-v3.2,5.0301,glm-5,2.9782,gemini-3.1-pro-preview/seed-2.0-pro +ReactionOptimisation_mit_case1_mixed,87.3082,98.66214557690091,gpt-5.4,98.6621,claude-opus-4.6,98.6041,deepseek-v3.2 +ReactionOptimisation_reizman_suzuki_pareto,63.5202,82.9901,glm-5,82.3427,claude-opus-4.6,82.24612252072882,gpt-5.4 +ReactionOptimisation_snar_multiobjective,57.5234,100.0,gpt-5.4,87.3657,claude-opus-4.6,82.7881,deepseek-v3.2 +Robotics_DynamicObstacleAvoidanceNavigation,0.0722,0.086,claude-opus-4.6,0.08571428571428559,gpt-5.4,0.0857,glm-5 +Robotics_PIDTuning,0.0366,0.1632,claude-opus-4.6,0.1585,grok-4.20,0.1521,gemini-3.1-pro-preview +Robotics_QuadrupedGaitOptimization,0.0218,0.1085,glm-5,0.0749,deepseek-v3.2,0.0232,qwen3-coder-next +Robotics_RobotArmCycleTimeOptimization,0.2922,0.4356212836221511,gpt-5.4,0.4305,gemini-3.1-pro-preview,0.4219,glm-5 +Robotics_UAVInspectionCoverageWithWind,28.8519,55.9109,grok-4.20,38.8024,deepseek-v3.2,35.1121,glm-5 +SingleCellAnalysis_predict_modality,0.5467,1.0,gpt-5.4,0.5467,claude-opus-4.6/deepseek-v3.2/gemini-3.1-pro-preview/glm-5/grok-4.20/qwen3-coder-next/seed-2.0-pro,0.5467,claude-opus-4.6/deepseek-v3.2/gemini-3.1-pro-preview/glm-5/grok-4.20/qwen3-coder-next/seed-2.0-pro +StructuralOptimization_ISCSO2015,-5401.589,-968.4567,claude-opus-4.6,-1120.212,deepseek-v3.2,-1139.3354,glm-5 +StructuralOptimization_ISCSO2023,-77813242.9,-16477799.48,claude-opus-4.6,-17840974.17,glm-5,-20092179.33,gemini-3.1-pro-preview +StructuralOptimization_TopologyOptimization,-195.9153,-185.7983,grok-4.20,-188.4673,glm-5,-189.3039,gemini-3.1-pro-preview +SustainableDataCenterControl_hand_written_control,8.3294,30.1873,qwen3-coder-next,29.2868,seed-2.0-pro,21.5657,claude-opus-4.6 +WirelessChannelSimulation_HighReliableSimulation,192.5193,304.0437,seed-2.0-pro,292.3228,claude-opus-4.6,291.9451,deepseek-v3.2 diff --git a/leaderboard/score_submission.py b/leaderboard/score_submission.py new file mode 100644 index 00000000..1c25561e --- /dev/null +++ b/leaderboard/score_submission.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +"""Score a submission against the frozen Frontier-Eng Medal podium. + +The gold/silver/bronze baselines are frozen at the v1 snapshot (2026-04-14) and +shipped in ``medal_podium.csv``. This script takes a new model's best-feasible +score on each task and reports its Medal Score, so anyone can be scored against +the released benchmark without rerunning the reference models. + +Usage +----- + python leaderboard/score_submission.py [--verbose] + +Submission CSV format (header required): two columns, ``Task,Score``, one row +per task, using the task names from ``medal_podium.csv`` (e.g. ``JobShop_abz``). +Higher score is better on every task. Missing tasks score 0. See +``submission_example.csv`` (the claude-opus-4.6 column) for a working example; +scoring it reproduces its leaderboard line (Medal v1 = 0.490, v1-lite = 0.501). + +Metric +------ +On each task a submission earns 1.00 / 0.67 / 0.33 for reaching the gold / +silver / bronze score, else 0. The Medal Score is the mean of this credit, +normalized to [0, 1], reported on the full v1 set (47 tasks) and the v1-lite +subset (10 tasks). +""" + +import argparse +import csv +from pathlib import Path + +HERE = Path(__file__).resolve().parent +GOLD, SILVER, BRONZE = 1.00, 0.67, 0.33 + +# v1-lite: 10-task representative subset (frontier_eval/conf/batch/v1_lite.yaml). +V1_LITE = { + "QuantumComputing_task_01_routing_qftentangled", "ComputerSystems_MallocLab", + "JobShop_abz", "InventoryOptimization_disruption_eoqd", + "EnergyStorage_BatteryFastChargingSPMe", "Robotics_RobotArmCycleTimeOptimization", + "Optics_holographic_multiplane_focusing", "WirelessChannelSimulation_HighReliableSimulation", + "ReactionOptimisation_snar_multiobjective", "StructuralOptimization_TopologyOptimization", +} + + +def load_podium(path): + """task -> (gold, silver, bronze) thresholds (higher is better).""" + podium = {} + with open(path, encoding="utf-8-sig") as f: + for row in csv.DictReader(f): + podium[row["Task"]] = ( + float(row["Gold"]), float(row["Silver"]), float(row["Bronze"])) + return podium + + +def load_submission(path): + """task -> score. Accepts a 'Task,Score' header or any two-column CSV.""" + scores = {} + with open(path, encoding="utf-8-sig") as f: + reader = csv.reader(f) + first = next(reader) + if not (first[1].strip().lower() in ("score", "best", "value")): + f.seek(0) # no recognizable header -> treat all rows as data + reader = csv.reader(f) + for row in reader: + if len(row) < 2 or not row[0].strip(): + continue + try: + scores[row[0].strip()] = float(row[1]) + except ValueError: + continue # skip header/garbage rows + return scores + + +def tier(score, gold, silver, bronze): + if score >= gold: + return GOLD, "gold" + if score >= silver: + return SILVER, "silver" + if score >= bronze: + return BRONZE, "bronze" + return 0.0, None + + +def score(podium, submission, verbose=False): + per_task = {} + counts = {"gold": 0, "silver": 0, "bronze": 0} + missing = [] + for task, (g, s, b) in podium.items(): + if task not in submission: + per_task[task] = 0.0 + missing.append(task) + continue + pts, name = tier(submission[task], g, s, b) + per_task[task] = pts + if name: + counts[name] += 1 + medal_v1 = sum(per_task.values()) / len(podium) + lite = [t for t in podium if t in V1_LITE] + medal_lite = sum(per_task[t] for t in lite) / len(lite) + + print(f"Medal Score (v1, 47 tasks) : {medal_v1:.3f}" + f" (gold {counts['gold']}, silver {counts['silver']}, bronze {counts['bronze']})") + print(f"Medal Score (v1-lite, 10 tasks) : {medal_lite:.3f}") + if missing: + print(f"\n[warn] {len(missing)} task(s) absent from submission (scored 0): " + f"{', '.join(missing[:5])}{' ...' if len(missing) > 5 else ''}") + if verbose: + print("\nper-task credit:") + for task in podium: + print(f" {per_task[task]:.2f} {task}") + return medal_v1, medal_lite + + +def main(): + ap = argparse.ArgumentParser(description="Score a submission against the frozen Medal podium.") + ap.add_argument("submission", help="CSV with columns Task,Score (one row per task)") + ap.add_argument("--podium", default=str(HERE / "medal_podium.csv"), + help="frozen gold/silver/bronze baselines (default: leaderboard/medal_podium.csv)") + ap.add_argument("--verbose", action="store_true", help="print per-task medal credit") + args = ap.parse_args() + score(load_podium(args.podium), load_submission(args.submission), args.verbose) + + +if __name__ == "__main__": + main() diff --git a/leaderboard/submission_example.csv b/leaderboard/submission_example.csv new file mode 100644 index 00000000..82f4c1a8 --- /dev/null +++ b/leaderboard/submission_example.csv @@ -0,0 +1,48 @@ +Task,Score +Aerodynamics_CarAerodynamicsSensing,0.9624 +Astrodynamics_MannedLunarLanding,6027.3126 +ComputerSystems_MallocLab,96 +Cryptographic_AES-128,11.8617 +Cryptographic_SHA-256,16.7955 +Cryptographic_SHA3-256,17.4003 +EnergyStorage_BatteryFastChargingProfile,120.8025 +EnergyStorage_BatteryFastChargingSPMe,71.8225 +EngDesign,1.3571 +InventoryOptimization_disruption_eoqd,0.6473 +InventoryOptimization_finite_horizon_dp,0.9596 +InventoryOptimization_general_meio,0.9929 +InventoryOptimization_joint_replenishment,0.8822 +InventoryOptimization_tree_gsm_safety_stock,0.75 +JobShop_abz,96.1035 +JobShop_swv,89.4966 +JobShop_ta,90.8322 +KernelEngineering_FlashAttention,983.5001 +KernelEngineering_MLA,1000.3859 +KernelEngineering_TriMul,357.1636 +Optics_adaptive_fault_tolerant_fusion,0.6398 +Optics_adaptive_temporal_smooth_control,0.8419 +Optics_fiber_guardband_spectrum_packing,0.6692 +Optics_fiber_mcs_power_scheduling,0.6542 +Optics_fiber_wdm_channel_power_allocation,0.6675 +Optics_holographic_multifocus_power_ratio,0.8072 +Optics_holographic_multiplane_focusing,0.6002 +Optics_phase_dammann_uniform_orders,99.7995 +Optics_phase_fourier_pattern_holography,82.1276 +PyPortfolioOpt_robust_mvo_rebalance,99.9946 +QuantumComputing_task_01_routing_qftentangled,5.0479 +QuantumComputing_task_02_clifford_t_synthesis,1.6633 +QuantumComputing_task_03_cross_target_qaoa,2.5781 +ReactionOptimisation_mit_case1_mixed,98.6621 +ReactionOptimisation_reizman_suzuki_pareto,82.3427 +ReactionOptimisation_snar_multiobjective,87.3657 +Robotics_DynamicObstacleAvoidanceNavigation,0.086 +Robotics_PIDTuning,0.1632 +Robotics_QuadrupedGaitOptimization,0.0219 +Robotics_RobotArmCycleTimeOptimization,0.4158 +Robotics_UAVInspectionCoverageWithWind,28.8519 +SingleCellAnalysis_predict_modality,0.5467 +StructuralOptimization_ISCSO2015,-968.4567 +StructuralOptimization_ISCSO2023,-16477799.48 +StructuralOptimization_TopologyOptimization,-190.1498 +SustainableDataCenterControl_hand_written_control,21.5657 +WirelessChannelSimulation_HighReliableSimulation,292.3228 From 9484de352a54394feb0177a387ee7c33270a6da6 Mon Sep 17 00:00:00 2001 From: han Date: Sun, 12 Jul 2026 21:44:14 +0800 Subject: [PATCH 2/3] feat(ComputerSystems): add IndexOptimization task --- .gitignore | 3 + .../IndexOptimization/README.md | 64 ++ .../IndexOptimization/README_zh-CN.md | 38 + .../ComputerSystems/IndexOptimization/Task.md | 83 +++ .../IndexOptimization/Task_zh-CN.md | 53 ++ .../IndexOptimization/baseline/heuristic.py | 130 ++++ .../IndexOptimization/baseline/result_log.txt | 36 + .../IndexOptimization/data/raw_task.json | 61 ++ .../data/tpch_sf1/gen_data.sh | 114 +++ .../data/tpch_sf1/queries/tpch_q1.json | 10 + .../data/tpch_sf1/queries/tpch_q10.json | 19 + .../data/tpch_sf1/queries/tpch_q12.json | 19 + .../data/tpch_sf1/queries/tpch_q3.json | 17 + .../data/tpch_sf1/queries/tpch_q5.json | 22 + .../data/tpch_sf1/queries/tpch_q6.json | 15 + .../data/tpch_sf1/schema.sql | 95 +++ .../frontier_eval/agent_files.txt | 10 + .../frontier_eval/artifact_files.txt | 2 + .../frontier_eval/candidate_destination.txt | 1 + .../frontier_eval/constraints.txt | 13 + .../frontier_eval/copy_files.txt | 1 + .../frontier_eval/eval_command.txt | 1 + .../frontier_eval/eval_cwd.txt | 1 + .../frontier_eval/initial_program.txt | 1 + .../frontier_eval/readonly_files.txt | 4 + .../references/constants.json | 23 + .../IndexOptimization/scripts/init.py | 92 +++ .../verification/docker/Dockerfile | 7 + .../verification/evaluator.py | 672 ++++++++++++++++++ .../verification/requirements.txt | 1 + .../task/database_index_optimization.yaml | 5 + 31 files changed, 1613 insertions(+) create mode 100644 benchmarks/ComputerSystems/IndexOptimization/README.md create mode 100644 benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md create mode 100644 benchmarks/ComputerSystems/IndexOptimization/Task.md create mode 100644 benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md create mode 100644 benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py create mode 100644 benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/copy_files.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_cwd.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt create mode 100644 benchmarks/ComputerSystems/IndexOptimization/references/constants.json create mode 100644 benchmarks/ComputerSystems/IndexOptimization/scripts/init.py create mode 100644 benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile create mode 100644 benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py create mode 100644 benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt create mode 100644 frontier_eval/conf/task/database_index_optimization.yaml diff --git a/.gitignore b/.gitignore index bf3b45cf..59782e89 100644 --- a/.gitignore +++ b/.gitignore @@ -56,3 +56,6 @@ debug-*.log **/outputs/ **/artifacts/ **/last_eval.json + +# IndexOptimization generated data +benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/tpch_sf1.dump diff --git a/benchmarks/ComputerSystems/IndexOptimization/README.md b/benchmarks/ComputerSystems/IndexOptimization/README.md new file mode 100644 index 00000000..614c0d39 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/README.md @@ -0,0 +1,64 @@ +# IndexOptimization — PostgreSQL Index Selection + +Optimize index configuration for a TPC-H SF1 analytical workload on PostgreSQL 16. + +## Benchmark ID + +`ComputerSystems/IndexOptimization` + +## Task + +Given a PostgreSQL database with TPC-H schema and a mixed analytical SQL workload, find a high-quality B-tree index configuration that improves query performance compared to a heuristic baseline, subject to index count and storage constraints. + +## Economic Relevance + +Database index optimization is a core database administration task with direct operational impact: +- PostgreSQL is one of the most widely deployed open-source databases in production, used by enterprises across finance, e-commerce, logistics, and analytics. +- Unoptimized indexes cause unnecessary storage costs, slow down write operations, and degrade query performance — a poorly indexed database can be 10-100x slower than a well-tuned one. +- Automated index selection reduces the need for manual DBA tuning, which is both expensive and error-prone at scale. +- This benchmark uses TPC-H, the industry-standard decision-support workload, making results transferable to real-world analytical processing environments. + +## Workload + +6 TPC-H SF1 queries covering multi-table JOINs, aggregation, range filters, sorting, and sequential scans. + +## Environment + +Requires Docker: + +```bash +docker build -t frontier-pg-index:latest verification/docker/ +``` + +## Quick Run + +```bash +python scripts/init.py --input data/raw_task.json --output outputs/candidate.json +python verification/evaluator.py scripts/init.py +``` + +## First-Time Setup + +Generate TPC-H SF1 data (required once before running the evaluator): + +```bash +bash data/tpch_sf1/gen_data.sh +``` + +## Unified Task + +```bash +python -m frontier_eval \ + task=unified \ + task.benchmark=ComputerSystems/IndexOptimization \ + task.runtime.isolation_mode=docker \ + task.runtime.docker_image=frontier-pg-index:latest \ + algorithm=openevolve \ + algorithm.iterations=0 +``` + +## Algorithm Agnostic + +This benchmark is algorithm-agnostic. Rule-based, ML-based, LLM-based, and search-based approaches are all welcome. + +Inspired by research on automated index recommendation including LLMIA (Zhao et al., arXiv:2503.07884, 2025). diff --git a/benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md b/benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md new file mode 100644 index 00000000..e64aaaba --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md @@ -0,0 +1,38 @@ +# IndexOptimization — PostgreSQL 索引优化 + +对 TPC-H SF1 分析型 workload，在 PostgreSQL 16 上优化索引配置。 + +## 统一 benchmark ID + +`ComputerSystems/IndexOptimization` + +## 任务描述 + +给定一个 PostgreSQL 数据库（TPC-H schema）和一个分析型 SQL workload，在索引数量和存储空间约束下，找到比 heuristic 基线更好的 B-tree 索引配置。 + +## 环境准备 + +```bash +docker build -t frontier-pg-index:latest verification/docker/ +``` + +## 快速运行 + +```bash +python scripts/init.py --input data/raw_task.json --output outputs/candidate.json +python verification/evaluator.py scripts/init.py +``` + +## Unified task 运行 + +```bash +python -m frontier_eval \ + task=unified \ + task.benchmark=ComputerSystems/IndexOptimization \ + task.runtime.isolation_mode=docker \ + task.runtime.docker_image=frontier-pg-index:latest \ + algorithm=openevolve \ + algorithm.iterations=0 +``` + +本 benchmark 对算法无关，基于规则的、ML-based、LLM-based、搜索算法均可参与。 diff --git a/benchmarks/ComputerSystems/IndexOptimization/Task.md b/benchmarks/ComputerSystems/IndexOptimization/Task.md new file mode 100644 index 00000000..a44d3937 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/Task.md @@ -0,0 +1,83 @@ +# IndexOptimization + +## Overview + +Find high-quality B-tree index configurations for a PostgreSQL database serving a mixed analytical SQL workload (TPC-H SF1). The goal is to improve query execution time under index count and storage constraints, compared to a heuristic baseline. + +## Input + +The candidate script (`scripts/init.py`) reads the task configuration from `--input` and writes output to `--output`. The input provides: + +- **Workload queries**: SQL queries with structured metadata (tables, filters, JOIN conditions) +- **Database schema**: Table definitions with column types, row counts, and existing indexes (primary keys) +- **Constraints**: `max_indexes` (10), `max_storage_mb` (500) + +## Output + +```json +{ + "indexes": [ + {"table": "orders", "columns": ["o_orderdate", "o_custkey"], "method": "btree"} + ] +} +``` + +## Scoring + +| Metric | Description | +|--------|-------------| +| `no_index_time_s` | Total query time without indexes (reference) | +| `baseline_time_s` | Total query time with heuristic indexes | +| `candidate_time_s` | Total query time with candidate indexes | +| `speedup` | baseline_time / candidate_time | +| `combined_score` | log2(speedup) × (1 - storage_penalty - count_penalty) | + +**Hard constraints**: All queries must return identical results; ≤10 indexes; ≤500 MB storage. + +## Workload Queries + +| ID | Pattern | Tables | Index optimization target | +|----|---------|--------|--------------------------| +| Q1 | Single-table aggregate | lineitem | Sequential scan test | +| Q3 | Join + filter + aggregate + order | customer, orders, lineitem | Multi-join, range filter | +| Q5 | Multi-way join + aggregate | 6 tables | Star schema join | +| Q6 | Single-table range filter + aggregate | lineitem | Selective filter | +| Q10 | Join + aggregate + order | customer, orders, lineitem, nation | Join + filter | +| Q12 | Join + dual-condition filter + aggregate | orders, lineitem | Filter selectivity | + +## Constraints + +1. Only modify `scripts/init.py` — the ONLY editable file. +2. Keep CLI contract: `--input` and `--output`. +3. Output must contain an `indexes` list. +4. Each index must specify table, columns (list), and method ("btree"). +5. Maximum 10 indexes per submission. +6. Total index storage must not exceed 500 MB. +7. All queries must return identical results before and after indexing. +8. Do not recommend indexes already covered by primary keys (listed in schema existing_indexes). + +## Economic Relevance + +Database index optimization directly affects production system performance and operational cost: +- **Query performance**: A well-chosen index can reduce query time from minutes to milliseconds in analytical workloads. +- **Storage cost**: Each unnecessary index consumes disk space and memory for caching. At TPC-H SF1 scale, an unused index costs ~10-100 MB. +- **Write overhead**: Indexes slow down INSERT/UPDATE/DELETE operations. In read-write mixed workloads, excess indexes degrade overall throughput. +- **Operational complexity**: Production databases often have hundreds of tables. Manual index tuning does not scale — automated index selection is a recognized industry need. + +This benchmark evaluates an agent's ability to make these engineering trade-offs: improving read performance while respecting storage and count constraints. + +## Optimization Directions + +Agents can explore the following strategies to find better index configurations: + +1. **Identify high-selectivity filters**: Indexing columns used in WHERE clauses with high selectivity (e.g., `o_orderdate`, `l_shipdate`) can significantly reduce scan ranges. + +2. **Covering indexes for JOIN columns**: Indexes on foreign key columns (e.g., `o_custkey`, `l_orderkey`) can accelerate hash join probe phases. + +3. **Avoid over-indexing**: Unused indexes incur storage and maintenance costs. The scoring formula penalizes both storage ratio and index count — more indexes do not always yield better scores. + +4. **Recognize sequential-scan scenarios**: Queries like Q1 (aggregating most of a large table) may not benefit from indexes. Over-indexing such queries adds cost without benefit. + +5. **Multi-column indexes**: A composite index (e.g., `(o_orderdate, o_custkey)`) can serve multiple query clauses simultaneously, potentially replacing several single-column indexes. + +6. **Analyze Q1's behavior**: Q1 performs a full table scan on `lineitem`. Indexes do not help this query but consume storage. The scoring formula accounts for storage cost — a good strategy should recognize when not to index. diff --git a/benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md b/benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md new file mode 100644 index 00000000..367449f1 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md @@ -0,0 +1,53 @@ +# IndexOptimization - 数据库索引优化 + +## 概述 + +在索引数量和存储空间约束下，为 PostgreSQL 数据库上的分析型 SQL workload 找到高质量的 B-tree 索引配置，以优化查询执行时间。 + +## 输入 + +candidate 脚本通过 `--input` 读取任务配置，将输出写入 `--output`。输入包含： + +- **查询 workload**：SQL 查询及结构化 metadata（涉及表、过滤条件、JOIN 条件） +- **数据库 schema**：表定义、列类型、行数、已有索引（主键） +- **约束**：`max_indexes` (10)、`max_storage_mb` (500) + +## 输出 + +```json +{ + "indexes": [ + {"table": "orders", "columns": ["o_orderdate", "o_custkey"], "method": "btree"} + ] +} +``` + +## 评分 + +| 指标 | 说明 | +|------|------| +| `baseline_time_s` | heuristic 索引下的总查询时间 | +| `candidate_time_s` | candidate 索引下的总查询时间 | +| `combined_score` | log2(speedup) × (1 - 存储惩罚 - 数量惩罚) | + +## 约束 + +1. 只修改 `scripts/init.py` +2. 保持 CLI 接口不变（`--input` 和 `--output`） +3. 输出必须包含 `indexes` 列表 +4. 每个索引需指定 table、columns 和 method +5. 最多 10 个索引 +6. 索引空间不超过 500 MB +7. 索引前后所有查询结果必须一致 +8. 不要推荐主键中已存在的索引 + +## 优化方向建议 + +Agent 可以尝试以下策略来找到更好的索引配置： + +1. **高选择性过滤列**：对 WHERE 子句中选择性高的列（如 `o_orderdate`、`l_shipdate`）建索引，可显著缩小扫描范围 +2. **JOIN 列覆盖索引**：对外键列（如 `o_custkey`、`l_orderkey`）建索引，可加速哈希连接的探测阶段 +3. **避免过度索引**：未被使用的索引会产生存储和维护开销。评分公式中存储惩罚和数量惩罚共同作用——更多的索引不一定带来更高的分数 +4. **识别全表扫描场景**：Q1 对大表大部分行做聚合查询，索引无法帮助此类查询。过度建索引只会增加成本 +5. **多列索引**：组合索引（如 `(o_orderdate, o_custkey)`）可同时服务于多个查询子句，可能替代多个单列索引 +6. **分析 Q1 的特性**：Q1 对 `lineitem` 做全表扫描，索引对此查询无帮助但消耗存储。评分公式考虑了存储成本，好的策略应能识别何时不应建索引 diff --git a/benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py b/benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py new file mode 100644 index 00000000..2f19c605 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py @@ -0,0 +1,130 @@ +""" +Heuristic index recommendation baseline. + +Strategy: +1. Collect all WHERE filter columns and JOIN columns from workload queries. +2. Remove columns already covered by existing indexes (primary keys). +3. Count column frequency across queries. +4. Create single-column B-tree indexes on the most frequent columns. +5. Stop before exceeding max_indexes. +""" + +from __future__ import annotations + +import json +from collections import Counter +from pathlib import Path +from typing import Any + + +def load_json(path: str | Path) -> dict[str, Any]: + with Path(path).open() as f: + return json.load(f) + + +def _resolve_queries(task_dir: Path, raw: dict[str, Any]) -> list[dict[str, Any]]: + queries: list[dict[str, Any]] = [] + for ref in raw.get("queries", []): + qpath = (task_dir / ref).resolve() + queries.append(load_json(qpath)) + return queries + + +def _flatten_filter_columns(metadata: dict) -> list[tuple[str, str]]: + """Extract (table, column) pairs from filter conditions.""" + result: list[tuple[str, str]] = [] + for f in metadata.get("filters", []): + tbl = f.get("table", "") + col = f.get("column", "") + if tbl and col: + result.append((tbl, col)) + return result + + +def _flatten_join_columns(metadata: dict) -> list[tuple[str, str]]: + """Extract (table, column) pairs from JOIN conditions (both sides).""" + result: list[tuple[str, str]] = [] + for jc in metadata.get("join_conditions", []): + for side in [jc.get("left", []), jc.get("right", [])]: + if len(side) >= 2: + result.append((side[0], side[1])) + return result + + +def _columns_in_existing_index(table: str, schema: dict[str, Any]) -> set[str]: + """Return set of column names already covered by existing indexes on this table.""" + covered: set[str] = set() + table_info = schema.get(table, {}) + for idx in table_info.get("existing_indexes", []): + for col in idx.get("columns", []): + covered.add(col) + return covered + + +def recommend_indexes( + queries: list[dict[str, Any]], + schema: dict[str, Any], + constraints: dict[str, Any], +) -> list[dict[str, Any]]: + max_indexes = constraints.get("max_indexes", 10) + + # Collect candidate columns from filters and joins + candidates: list[tuple[str, str]] = [] + for q in queries: + meta = q.get("metadata", {}) + candidates.extend(_flatten_filter_columns(meta)) + candidates.extend(_flatten_join_columns(meta)) + + # Count frequency and deduplicate + freq = Counter(candidates) + + # Sort by frequency (descending), then by table name for determinism + ranked = sorted(freq.items(), key=lambda x: (-x[1], x[0][0], x[0][1])) + + # Build index list, skipping columns already covered by existing indexes + result: list[dict[str, Any]] = [] + seen_pairs: set[tuple[str, str]] = set() + + for (table, column), _count in ranked: + if len(result) >= max_indexes: + break + if (table, column) in seen_pairs: + continue + # Skip if column already has an index + existing = _columns_in_existing_index(table, schema) + if column in existing: + continue + seen_pairs.add((table, column)) + result.append({"table": table, "columns": [column], "method": "btree"}) + + return result + + +def main() -> None: + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--input", required=True) + parser.add_argument("--output", required=True) + args = parser.parse_args() + + input_path = Path(args.input).resolve() + task_dir = input_path.parent.parent + raw = load_json(input_path) + queries = _resolve_queries(task_dir, raw) + schema = raw.get("schema", {}) + constraints = raw.get("constraints", {}) + + indexes = recommend_indexes(queries, schema, constraints) + + output_path = Path(args.output) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text( + json.dumps({"indexes": indexes}, indent=2), encoding="utf-8" + ) + print(f"heuristic indexes: {len(indexes)}") + for idx in indexes: + print(f" {idx['table']}({', '.join(idx['columns'])})") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt b/benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt new file mode 100644 index 00000000..e16311a1 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt @@ -0,0 +1,36 @@ +IndexOptimization baseline (heuristic) result log +================================================ +Date: 2026-07-11 +Evaluator: verification/evaluator.py (evaluate() function) +Candidate: scripts/init.py (baseline: returns empty index list) +Environment: Docker (frontier-pg-index:latest), TPC-H SF1 + +Results: +-------- +valid: 1.0 +combined_score: 0.632 +no_index_time_s: 7.0 (no indexes, reference only) +baseline_time_s: 9.8 (heuristic: 10 single-column indexes) +candidate_time_s: 5.9 (candidate: 0 additional indexes) +n_indexes: 0 (beyond existing primary keys) +storage_mb: 237.6 (primary key indexes only) +data_size_mb: 1312.9 (TPC-H SF1 ~1.3 GB) +correct: 1.0 (all query results match) +speedup: 1.67 (candidate vs heuristic) + +Observations: +------------- +1. The heuristic baseline (10 indexes) is SLOWER than no-index (9.8s vs 7.0s). + This demonstrates that naive single-column indexing on filter/join columns + can degrade performance through unnecessary overhead. +2. The candidate (0 extra indexes) outperforms heuristic by 1.67x. +3. A good agent should find a configuration that: + - Selectively indexes high-impact columns (e.g., l_shipdate, o_orderdate) + - Avoids indexing columns where sequential scans are preferable (Q1) + - Stays within the 10-index and 500 MB constraints + +Heuristic index list (10 single-column): + orders(o_orderdate), lineitem(l_shipdate), orders(o_custkey), + customer(c_nationkey), lineitem(l_receiptdate), supplier(s_nationkey), + customer(c_mktsegment), lineitem(l_commitdate), lineitem(l_discount), + lineitem(l_quantity) diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json b/benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json new file mode 100644 index 00000000..f0e0e701 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json @@ -0,0 +1,61 @@ +{ + "benchmark_id": "index_optimization", + "description": "PostgreSQL index optimization on TPC-H SF1 workload", + "queries": [ + "data/tpch_sf1/queries/tpch_q1.json", + "data/tpch_sf1/queries/tpch_q3.json", + "data/tpch_sf1/queries/tpch_q5.json", + "data/tpch_sf1/queries/tpch_q6.json", + "data/tpch_sf1/queries/tpch_q10.json", + "data/tpch_sf1/queries/tpch_q12.json" + ], + "schema": { + "nation": { + "columns": [{"name": "n_nationkey", "type": "integer"}], + "row_count": 25, + "existing_indexes": [{"columns": ["n_nationkey"], "type": "primary"}] + }, + "region": { + "columns": [{"name": "r_regionkey", "type": "integer"}], + "row_count": 5, + "existing_indexes": [{"columns": ["r_regionkey"], "type": "primary"}] + }, + "part": { + "columns": [{"name": "p_partkey", "type": "integer"}], + "row_count": 200000, + "existing_indexes": [{"columns": ["p_partkey"], "type": "primary"}] + }, + "supplier": { + "columns": [{"name": "s_suppkey", "type": "integer"}], + "row_count": 10000, + "existing_indexes": [{"columns": ["s_suppkey"], "type": "primary"}] + }, + "partsupp": { + "columns": [{"name": "ps_partkey", "type": "integer"}, {"name": "ps_suppkey", "type": "integer"}], + "row_count": 800000, + "existing_indexes": [{"columns": ["ps_partkey", "ps_suppkey"], "type": "primary"}] + }, + "customer": { + "columns": [{"name": "c_custkey", "type": "integer"}], + "row_count": 150000, + "existing_indexes": [{"columns": ["c_custkey"], "type": "primary"}] + }, + "orders": { + "columns": [{"name": "o_orderkey", "type": "bigint"}], + "row_count": 1500000, + "existing_indexes": [{"columns": ["o_orderkey"], "type": "primary"}] + }, + "lineitem": { + "columns": [{"name": "l_orderkey", "type": "bigint"}, {"name": "l_linenumber", "type": "integer"}], + "row_count": 6001215, + "existing_indexes": [{"columns": ["l_orderkey", "l_linenumber"], "type": "primary"}] + } + }, + "constraints": { + "max_indexes": 10, + "max_storage_mb": 500, + "statement_timeout_s": 60, + "query_repeats": 3, + "warmup_rounds": 2 + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh new file mode 100644 index 00000000..8b415fad --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# Generate TPC-H SF1 data for IndexOptimization benchmark. +# +# Usage: +# bash data/tpch_sf1/gen_data.sh +# +# This script: +# 1. Checks that Docker and the frontier-pg-index image are available +# 2. Clones dbgen and generates TPC-H SF1 data files +# 3. Starts a PostgreSQL container and loads the data +# 4. Creates a pg_dump file for fast evaluator restore +# 5. Cleans up + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +DUMP_FILE="${SCRIPT_DIR}/tpch_sf1.dump" +SCHEMA_FILE="${SCRIPT_DIR}/schema.sql" +DBGEN_DIR="/tmp/tpch_dbgen_index_opt" + +# Prerequisites +if ! command -v docker &>/dev/null; then + echo "Error: docker is required but not found." + exit 1 +fi + +if ! docker image inspect frontier-pg-index:latest &>/dev/null; then + echo "Building frontier-pg-index:latest image..." + REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" + docker build -t frontier-pg-index:latest "${REPO_ROOT}/verification/docker/" +fi + +# Clone or update dbgen +if [ -d "${DBGEN_DIR}" ]; then + echo "dbgen already exists at ${DBGEN_DIR}" +else + echo "Downloading TPC-H dbgen..." + if command -v git &>/dev/null; then + git clone --depth 1 https://github.com/electrum/tpch-dbgen.git "${DBGEN_DIR}" + else + mkdir -p "${DBGEN_DIR}" + cd "${DBGEN_DIR}" && curl -sL "https://github.com/electrum/tpch-dbgen/archive/refs/heads/master.tar.gz" | tar -xz --strip=1 + fi +fi + +# Compile dbgen +echo "Compiling dbgen..." +cd "${DBGEN_DIR}" +if [ ! -f dbgen ]; then + make -j4 2>&1 || { echo "dbgen compilation failed"; exit 1; } +fi + +# Generate data +echo "Generating TPC-H SF1 data files..." +./dbgen -s 1 -f +echo "Data files: $(ls *.tbl | wc -l) tables" + +# Start PostgreSQL container +echo "Starting PostgreSQL..." +CONTAINER_ID=$(docker run -d --rm \ + -e POSTGRES_PASSWORD=postgres \ + -v "${DBGEN_DIR}:/tpch_data" \ + -P frontier-pg-index:latest) + +# Wait for PostgreSQL to be ready +for i in $(seq 1 30); do + if docker exec "${CONTAINER_ID}" pg_isready -q 2>/dev/null; then + echo "PostgreSQL ready after ${i}s" + break + fi + sleep 1 + if [ "$i" -eq 30 ]; then + echo "Error: PostgreSQL did not start within 30s" + docker stop "${CONTAINER_ID}" >/dev/null + exit 1 + fi +done + +# Remove trailing pipe from .tbl files (TPC-H format compatibility) +for f in "${DBGEN_DIR}"/*.tbl; do + sed -i.bak 's/|$//' "$f" + rm -f "${f}.bak" +done + +# Create schema +echo "Creating database schema..." +docker exec -i "${CONTAINER_ID}" psql -U postgres < "${SCHEMA_FILE}" + +# Load data +echo "Loading TPC-H SF1 data (this may take a few minutes)..." +for tbl in nation region part supplier partsupp customer orders lineitem; do + echo -n " ${tbl}... " + docker exec -i "${CONTAINER_ID}" psql -U postgres -c \ + "\\copy ${tbl} FROM '/tpch_data/${tbl}.tbl' WITH DELIMITER '|' NULL ''" \ + 2>&1 | grep -c "^COPY" | xargs echo -n + echo " rows loaded" +done + +# Create dump +echo "Creating pg_dump archive..." +docker exec "${CONTAINER_ID}" pg_dump -U postgres -Fc -f /tmp/tpch_sf1.dump +docker cp "${CONTAINER_ID}:/tmp/tpch_sf1.dump" "${DUMP_FILE}" + +# Stop container +docker stop "${CONTAINER_ID}" >/dev/null + +echo "" +echo "==============================" +echo "TPC-H SF1 data generated successfully!" +echo "Dump file: ${DUMP_FILE}" +ls -lh "${DUMP_FILE}" +echo "" +echo "The evaluator will automatically use this dump on restore." +echo "==============================" diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json new file mode 100644 index 00000000..9c56466b --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json @@ -0,0 +1,10 @@ +{ + "id": "tpch_q1", + "sql": "SELECT l_returnflag, l_linestatus, SUM(l_quantity) AS sum_qty, SUM(l_extendedprice) AS sum_base_price, SUM(l_extendedprice * (1 - l_discount)) AS sum_disc_price, SUM(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, AVG(l_quantity) AS avg_qty, AVG(l_extendedprice) AS avg_price, AVG(l_discount) AS avg_disc, COUNT(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-09-02' GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus", + "metadata": { + "tables": ["lineitem"], + "filters": [{"table": "lineitem", "column": "l_shipdate", "operator": "<="}], + "aggregate": true, + "sort_columns": ["l_returnflag", "l_linestatus"] + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json new file mode 100644 index 00000000..7b2601fb --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json @@ -0,0 +1,19 @@ +{ + "id": "tpch_q10", + "sql": "SELECT c_custkey, c_name, SUM(l_extendedprice * (1 - l_discount)) AS revenue, c_acctbal, n_name, c_address, c_phone, c_comment FROM customer, orders, lineitem, nation WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate >= DATE '1993-10-01' AND o_orderdate < DATE '1994-01-01' AND l_returnflag = 'R' AND c_nationkey = n_nationkey GROUP BY c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment ORDER BY revenue DESC", + "metadata": { + "tables": ["customer", "orders", "lineitem", "nation"], + "join_conditions": [ + {"left": ["customer", "c_custkey"], "right": ["orders", "o_custkey"]}, + {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]}, + {"left": ["customer", "c_nationkey"], "right": ["nation", "n_nationkey"]} + ], + "filters": [ + {"table": "orders", "column": "o_orderdate", "operator": ">="}, + {"table": "orders", "column": "o_orderdate", "operator": "<"}, + {"table": "lineitem", "column": "l_returnflag", "operator": "="} + ], + "aggregate": true, + "sort_columns": ["revenue"] + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json new file mode 100644 index 00000000..a304db52 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json @@ -0,0 +1,19 @@ +{ + "id": "tpch_q12", + "sql": "SELECT l_shipmode, SUM(CASE WHEN o_orderpriority = '1-URGENT' OR o_orderpriority = '2-HIGH' THEN 1 ELSE 0 END) AS high_line_count, SUM(CASE WHEN o_orderpriority <> '1-URGENT' AND o_orderpriority <> '2-HIGH' THEN 1 ELSE 0 END) AS low_line_count FROM orders, lineitem WHERE o_orderkey = l_orderkey AND l_shipmode IN ('MAIL', 'SHIP') AND l_commitdate < l_receiptdate AND l_shipdate < l_commitdate AND l_receiptdate >= DATE '1994-01-01' AND l_receiptdate < DATE '1995-01-01' GROUP BY l_shipmode ORDER BY l_shipmode", + "metadata": { + "tables": ["orders", "lineitem"], + "join_conditions": [ + {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]} + ], + "filters": [ + {"table": "lineitem", "column": "l_shipmode", "operator": "in"}, + {"table": "lineitem", "column": "l_commitdate", "operator": "<"}, + {"table": "lineitem", "column": "l_shipdate", "operator": "<"}, + {"table": "lineitem", "column": "l_receiptdate", "operator": ">="}, + {"table": "lineitem", "column": "l_receiptdate", "operator": "<"} + ], + "aggregate": true, + "sort_columns": ["l_shipmode"] + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json new file mode 100644 index 00000000..9e059ed1 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json @@ -0,0 +1,17 @@ +{ + "id": "tpch_q3", + "sql": "SELECT l_orderkey, SUM(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer, orders, lineitem WHERE c_mktsegment = 'BUILDING' AND c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate < DATE '1995-03-15' GROUP BY l_orderkey, o_orderdate, o_shippriority ORDER BY revenue DESC, o_orderdate", + "metadata": { + "tables": ["customer", "orders", "lineitem"], + "join_conditions": [ + {"left": ["customer", "c_custkey"], "right": ["orders", "o_custkey"]}, + {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]} + ], + "filters": [ + {"table": "customer", "column": "c_mktsegment", "operator": "="}, + {"table": "orders", "column": "o_orderdate", "operator": "<"} + ], + "aggregate": true, + "sort_columns": ["revenue", "o_orderdate"] + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json new file mode 100644 index 00000000..fb452b53 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json @@ -0,0 +1,22 @@ +{ + "id": "tpch_q5", + "sql": "SELECT n_name, SUM(l_extendedprice * (1 - l_discount)) AS revenue FROM customer, orders, lineitem, supplier, nation, region WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND l_suppkey = s_suppkey AND c_nationkey = s_nationkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'ASIA' AND o_orderdate >= DATE '1994-01-01' AND o_orderdate < DATE '1995-01-01' GROUP BY n_name ORDER BY revenue DESC", + "metadata": { + "tables": ["customer", "orders", "lineitem", "supplier", "nation", "region"], + "join_conditions": [ + {"left": ["customer", "c_custkey"], "right": ["orders", "o_custkey"]}, + {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]}, + {"left": ["lineitem", "l_suppkey"], "right": ["supplier", "s_suppkey"]}, + {"left": ["customer", "c_nationkey"], "right": ["supplier", "s_nationkey"]}, + {"left": ["supplier", "s_nationkey"], "right": ["nation", "n_nationkey"]}, + {"left": ["nation", "n_regionkey"], "right": ["region", "r_regionkey"]} + ], + "filters": [ + {"table": "region", "column": "r_name", "operator": "="}, + {"table": "orders", "column": "o_orderdate", "operator": ">="}, + {"table": "orders", "column": "o_orderdate", "operator": "<"} + ], + "aggregate": true, + "sort_columns": ["revenue"] + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json new file mode 100644 index 00000000..118d52df --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json @@ -0,0 +1,15 @@ +{ + "id": "tpch_q6", + "sql": "SELECT SUM(l_extendedprice * l_discount) AS revenue FROM lineitem WHERE l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1995-01-01' AND l_discount BETWEEN 0.05 AND 0.07 AND l_quantity < 24", + "metadata": { + "tables": ["lineitem"], + "filters": [ + {"table": "lineitem", "column": "l_shipdate", "operator": ">="}, + {"table": "lineitem", "column": "l_shipdate", "operator": "<"}, + {"table": "lineitem", "column": "l_discount", "operator": "between"}, + {"table": "lineitem", "column": "l_quantity", "operator": "<"} + ], + "aggregate": true, + "sort_columns": [] + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql new file mode 100644 index 00000000..80568de8 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql @@ -0,0 +1,95 @@ +-- TPC-H SF1 Schema (8 tables) +-- Primary keys create implicit indexes tracked in raw_task.json existing_indexes + +CREATE TABLE nation ( + n_nationkey INTEGER NOT NULL, + n_name CHAR(25) NOT NULL, + n_regionkey INTEGER NOT NULL, + n_comment VARCHAR(152), + PRIMARY KEY (n_nationkey) +); + +CREATE TABLE region ( + r_regionkey INTEGER NOT NULL, + r_name CHAR(25) NOT NULL, + r_comment VARCHAR(152), + PRIMARY KEY (r_regionkey) +); + +CREATE TABLE part ( + p_partkey INTEGER NOT NULL, + p_name VARCHAR(55) NOT NULL, + p_mfgr CHAR(25) NOT NULL, + p_brand CHAR(10) NOT NULL, + p_type VARCHAR(25) NOT NULL, + p_size INTEGER NOT NULL, + p_container CHAR(10) NOT NULL, + p_retailprice DECIMAL(15,2) NOT NULL, + p_comment VARCHAR(23) NOT NULL, + PRIMARY KEY (p_partkey) +); + +CREATE TABLE supplier ( + s_suppkey INTEGER NOT NULL, + s_name CHAR(25) NOT NULL, + s_address VARCHAR(40) NOT NULL, + s_nationkey INTEGER NOT NULL, + s_phone CHAR(15) NOT NULL, + s_acctbal DECIMAL(15,2) NOT NULL, + s_comment VARCHAR(101) NOT NULL, + PRIMARY KEY (s_suppkey) +); + +CREATE TABLE partsupp ( + ps_partkey INTEGER NOT NULL, + ps_suppkey INTEGER NOT NULL, + ps_availqty INTEGER NOT NULL, + ps_supplycost DECIMAL(15,2) NOT NULL, + ps_comment VARCHAR(199) NOT NULL, + PRIMARY KEY (ps_partkey, ps_suppkey) +); + +CREATE TABLE customer ( + c_custkey INTEGER NOT NULL, + c_name VARCHAR(25) NOT NULL, + c_address VARCHAR(40) NOT NULL, + c_nationkey INTEGER NOT NULL, + c_phone CHAR(15) NOT NULL, + c_acctbal DECIMAL(15,2) NOT NULL, + c_mktsegment CHAR(10) NOT NULL, + c_comment VARCHAR(117) NOT NULL, + PRIMARY KEY (c_custkey) +); + +CREATE TABLE orders ( + o_orderkey BIGINT NOT NULL, + o_custkey INTEGER NOT NULL, + o_orderstatus CHAR(1) NOT NULL, + o_totalprice DECIMAL(15,2) NOT NULL, + o_orderdate DATE NOT NULL, + o_orderpriority CHAR(15) NOT NULL, + o_clerk CHAR(15) NOT NULL, + o_shippriority INTEGER NOT NULL, + o_comment VARCHAR(79) NOT NULL, + PRIMARY KEY (o_orderkey) +); + +CREATE TABLE lineitem ( + l_orderkey BIGINT NOT NULL, + l_partkey INTEGER NOT NULL, + l_suppkey INTEGER NOT NULL, + l_linenumber INTEGER NOT NULL, + l_quantity DECIMAL(15,2) NOT NULL, + l_extendedprice DECIMAL(15,2) NOT NULL, + l_discount DECIMAL(15,2) NOT NULL, + l_tax DECIMAL(15,2) NOT NULL, + l_returnflag CHAR(1) NOT NULL, + l_linestatus CHAR(1) NOT NULL, + l_shipdate DATE NOT NULL, + l_commitdate DATE NOT NULL, + l_receiptdate DATE NOT NULL, + l_shipinstruct CHAR(25) NOT NULL, + l_shipmode CHAR(10) NOT NULL, + l_comment VARCHAR(44) NOT NULL, + PRIMARY KEY (l_orderkey, l_linenumber) +); diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt new file mode 100644 index 00000000..8eb44074 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt @@ -0,0 +1,10 @@ +README.md +README_zh-CN.md +Task.md +Task_zh-CN.md +scripts/init.py +verification/evaluator.py +baseline/heuristic.py +data/raw_task.json +data/tpch_sf1/queries/ +references/constants.json diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..4fe6c4c1 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt @@ -0,0 +1,2 @@ +metrics.json +outputs/*.json diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt new file mode 100644 index 00000000..c4a5f13d --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt @@ -0,0 +1,13 @@ +IndexOptimization constraints: +1) Candidate file is `scripts/init.py` and must output JSON with: + - `indexes`: list of {"table": str, "columns": list[str], "method": "btree"} +2) SQL safety: + - Only B-tree index method is supported. + - Index columns must exist in the target table's schema. +3) Semantic correctness: + - All workload queries must return identical results before and after indexing. +4) Limits: + - max indexes = 10 + - max index storage = 500 MB +5) Read-only benchmark assets: + - verification/evaluator.py, data/, references/, baseline/. diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/copy_files.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/copy_files.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/copy_files.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt new file mode 100644 index 00000000..613443e7 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +{python} {benchmark}/verification/evaluator.py {candidate} diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_cwd.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_cwd.txt new file mode 100644 index 00000000..9c558e35 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_cwd.txt @@ -0,0 +1 @@ +. diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt new file mode 100644 index 00000000..b9411b3d --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +scripts/init.py diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..c4bb61c3 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt @@ -0,0 +1,4 @@ +verification/evaluator.py +data/ +references/ +baseline/ diff --git a/benchmarks/ComputerSystems/IndexOptimization/references/constants.json b/benchmarks/ComputerSystems/IndexOptimization/references/constants.json new file mode 100644 index 00000000..3dd76b32 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/references/constants.json @@ -0,0 +1,23 @@ +{ + "scoring": { + "storage_penalty_weight": 0.3, + "count_penalty_weight": 0.1, + "numeric_tolerance": 1e-6, + "parameter_notes": { + "storage_penalty_weight": "At 50% storage ratio (max allowed), penalizes score by 0.3. Chosen so that moderate storage use is acceptable but excessive indexing is discouraged.", + "count_penalty_weight": "At max 10 indexes, penalizes score by 0.1. Lower than storage_penalty to prioritize storage cost over index count.", + "numeric_tolerance": "Standard floating-point tolerance for result correctness comparison." + } + }, + "measurement": { + "query_repeats": 3, + "warmup_rounds": 2, + "statement_timeout_s": 60, + "single_threaded": true, + "parameter_notes": { + "query_repeats": "Median of 3 runs reduces timing noise without excessive wall-clock cost.", + "warmup_rounds": "2 warmup rounds stabilize buffer pool and query planning before measurement.", + "single_threaded": "SET max_parallel_workers_per_gather = 0 ensures reproducible single-threaded timing." + } + } +} diff --git a/benchmarks/ComputerSystems/IndexOptimization/scripts/init.py b/benchmarks/ComputerSystems/IndexOptimization/scripts/init.py new file mode 100644 index 00000000..95848322 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/scripts/init.py @@ -0,0 +1,92 @@ +# EVOLVE-BLOCK-START +"""Index Optimization candidate program — recommend PostgreSQL indexes for TPC-H workload.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path +from typing import Any + + +# DO NOT MODIFY: CLI contract +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Index Optimization candidate — recommend PostgreSQL indexes" + ) + parser.add_argument("--input", required=True, help="Path to raw_task.json") + parser.add_argument("--output", required=True, help="Output JSON path") + return parser.parse_args() + + +# DO NOT MODIFY: input loading +def load_json(path: str | Path) -> dict[str, Any]: + with Path(path).open() as f: + return json.load(f) + + +# DO NOT MODIFY: recursive resolve of query references +def _resolve_queries(task_dir: Path, raw: dict[str, Any]) -> list[dict[str, Any]]: + queries: list[dict[str, Any]] = [] + for ref in raw.get("queries", []): + qpath = (task_dir / ref).resolve() + q = load_json(qpath) + # Overlay benchmark_id context + q["__query_file__"] = str(qpath) + queries.append(q) + return queries + + +# MODIFIABLE: index recommendation strategy (core optimization target) +def recommend_indexes( + queries: list[dict[str, Any]], + schema: dict[str, Any], + constraints: dict[str, Any], +) -> list[dict[str, Any]]: + """ + Recommend a set of B-tree indexes to optimize query performance. + + Args: + queries: List of query dicts, each with {"id", "sql", "metadata"}. + schema: Dict mapping table name to {"columns", "row_count", "existing_indexes"}. + constraints: {"max_indexes": int, "max_storage_mb": int, ...} + + Returns: + List of index specs: [{"table": str, "columns": list[str], "method": "btree"}, ...] + """ + # Baseline: no additional indexes beyond existing primary keys + return [] + + +# DO NOT MODIFY: output format +def solve(raw: dict[str, Any], task_dir: Path) -> dict[str, Any]: + queries = _resolve_queries(task_dir, raw) + schema = raw.get("schema", {}) + constraints = raw.get("constraints", {}) + indexes = recommend_indexes(queries, schema, constraints) + return { + "indexes": indexes, + "benchmark_id": raw.get("benchmark_id", "index_optimization"), + } + + +# DO NOT MODIFY: entry point +def main() -> None: + args = _parse_args() + input_path = Path(args.input).resolve() + output_path = Path(args.output).resolve() + + task_dir = input_path.parent.parent # data/ -> task root + raw = load_json(input_path) + submission = solve(raw, task_dir) + + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(json.dumps(submission, indent=2), encoding="utf-8") + print(f"indexes: {len(submission.get('indexes', []))}") + print(f"submission: {output_path}") + + +if __name__ == "__main__": + main() +# EVOLVE-BLOCK-END diff --git a/benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile b/benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile new file mode 100644 index 00000000..29a83bd7 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile @@ -0,0 +1,7 @@ +FROM postgres:16 + +# Python runtime for evaluator scripts +RUN apt-get update && apt-get install -y python3 python3-pip python3-psycopg2 && rm -rf /var/lib/apt/lists/* + +# Default entry point: PostgreSQL +CMD ["postgres"] diff --git a/benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py b/benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py new file mode 100644 index 00000000..b11d8356 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py @@ -0,0 +1,672 @@ +"""Evaluator for PostgreSQL index optimization benchmark (TPC-H SF1 workload).""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import re +import shutil +import subprocess +import sys +import tempfile +import time +import traceback +from pathlib import Path +from typing import Any + +INVALID_COMBINED_SCORE = -1e18 + +# --------------------------------------------------------------------------- +# Path helpers +# --------------------------------------------------------------------------- + +def _is_repo_root(path: Path) -> bool: + return (path / "benchmarks").is_dir() and (path / "frontier_eval").is_dir() + + +def _find_repo_root() -> Path: + env_root = (os.environ.get("FRONTIER_ENGINEERING_ROOT") or "").strip() + if env_root: + cand = Path(env_root).expanduser().resolve() + if _is_repo_root(cand): + return cand + here = Path(__file__).resolve() + for parent in [here.parent, *here.parents]: + if _is_repo_root(parent): + return parent + return Path.cwd().resolve() + + +def _task_dir(repo_root: Path) -> Path: + return repo_root / "benchmarks" / "ComputerSystems" / "IndexOptimization" + + +# --------------------------------------------------------------------------- +# JSON helpers +# --------------------------------------------------------------------------- + +def load_json(path: str | Path) -> dict[str, Any]: + with Path(path).open() as f: + return json.load(f) + + +def dump_json(path: str | Path, payload: dict[str, Any]) -> None: + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("w") as f: + json.dump(payload, f, indent=2, sort_keys=True) + + +# --------------------------------------------------------------------------- +# PostgreSQL helpers +# --------------------------------------------------------------------------- + +def _pg_connect(port: int, dbname: str = "postgres") -> Any: + import psycopg2 + return psycopg2.connect( + host="localhost", port=port, dbname=dbname, + user="postgres", password="postgres", + ) + + +def _run_sql(conn: Any, sql: str, timeout_s: int = 60) -> list[tuple]: + with conn.cursor() as cur: + cur.execute(f"SET statement_timeout = '{timeout_s}s'") + cur.execute(sql) + if cur.description is not None: + return cur.fetchall() + return [] + + +def _table_exists(conn: Any, table: str) -> bool: + sql = "SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = %s)" + with conn.cursor() as cur: + cur.execute(sql, (table,)) + return cur.fetchone()[0] + + +def _column_exists(conn: Any, table: str, column: str) -> bool: + sql = "SELECT EXISTS (SELECT FROM information_schema.columns WHERE table_name = %s AND column_name = %s)" + with conn.cursor() as cur: + cur.execute(sql, (table, column)) + return cur.fetchone()[0] + + +def _get_index_size_mb(conn: Any) -> float: + sql = """ + SELECT COALESCE(SUM(pg_relation_size(indexrelid)), 0) / 1048576.0 + FROM pg_stat_user_indexes + """ + with conn.cursor() as cur: + cur.execute(sql) + return float(cur.fetchone()[0]) + + +def _get_data_size_mb(conn: Any) -> float: + sql = """ + SELECT COALESCE(SUM(pg_relation_size(relid)), 0) / 1048576.0 + FROM pg_stat_user_tables + """ + with conn.cursor() as cur: + cur.execute(sql) + return float(cur.fetchone()[0]) + + +def _quote_ident(name: str) -> str: + return f'"{name}"' + + +def _create_single_index(conn: Any, table: str, columns: list[str], method: str) -> None: + cols = ", ".join(_quote_ident(c) for c in columns) + idx_name = f"idx_{table}_{'_'.join(columns)}" + sql = f"CREATE INDEX IF NOT EXISTS {_quote_ident(idx_name)} ON {_quote_ident(table)} USING {method} ({cols})" + with conn.cursor() as cur: + cur.execute(sql) + conn.commit() + + +def _get_actual_indexes(conn: Any) -> list[dict[str, Any]]: + """Return list of actual indexes currently in the database.""" + sql = """ + SELECT schemaname, tablename, indexname, indexdef + FROM pg_indexes + WHERE schemaname = 'public' + ORDER BY tablename, indexname + """ + with conn.cursor() as cur: + cur.execute(sql) + rows = cur.fetchall() + result = [] + for row in rows: + result.append({ + "table": row[1], + "index_name": row[2], + "definition": row[3], + }) + return result + + +# --------------------------------------------------------------------------- +# Container management +# --------------------------------------------------------------------------- + +def _start_postgres(docker_image: str, data_dir: Path | None = None) -> tuple[str, int]: + """Start a PostgreSQL Docker container, return (container_id, port).""" + cmd = [ + "docker", "run", "-d", + "--rm", + "-e", "POSTGRES_PASSWORD=postgres", + "-e", "POSTGRES_DB=postgres", + "-P", # random port + ] + if data_dir is not None and data_dir.exists(): + cmd += ["-v", f"{data_dir.resolve()}:/var/lib/postgresql/data"] + cmd.append(docker_image) + + result = subprocess.run(cmd, capture_output=True, text=True, timeout=30) + if result.returncode != 0: + raise RuntimeError(f"Docker start failed: {result.stderr}") + + container_id = result.stdout.strip() + + # Get the mapped port + port_result = subprocess.run( + ["docker", "port", container_id, "5432"], + capture_output=True, text=True, timeout=10, + ) + if port_result.returncode != 0: + _stop_postgres(container_id) + raise RuntimeError(f"Failed to get port: {port_result.stderr}") + + port = int(port_result.stdout.strip().split(":")[-1]) + + # Wait for PostgreSQL to be ready + for _ in range(30): + try: + conn = _pg_connect(port) + conn.close() + return container_id, port + except Exception: + time.sleep(1) + + _stop_postgres(container_id) + raise RuntimeError("PostgreSQL did not become ready within 30s") + + +def _stop_postgres(container_id: str) -> None: + subprocess.run(["docker", "stop", container_id], capture_output=True, timeout=30) + + +# --------------------------------------------------------------------------- +# Data restore helper +# --------------------------------------------------------------------------- + +def _restore_dump(container_id: str, dump_path: Path) -> None: + """Copy and restore a pg_dump archive into a running PostgreSQL container.""" + subprocess.run( + f"docker cp {dump_path} {container_id}:/tmp/tpch_sf1.dump".split(), + capture_output=True, timeout=30, + ) + subprocess.run( + ["docker", "exec", "-i", container_id, + "pg_restore", "-U", "postgres", "-d", "postgres", + "-Fc", "--clean", "/tmp/tpch_sf1.dump"], + capture_output=True, text=True, timeout=300, + ) + + +# --------------------------------------------------------------------------- +# SQL loading helpers +# --------------------------------------------------------------------------- + +def _resolve_queries(task_dir: Path, raw: dict[str, Any]) -> list[dict[str, Any]]: + queries = [] + for ref in raw.get("queries", []): + qpath = (task_dir / ref).resolve() + q = load_json(qpath) + queries.append(q) + return queries + + +# --------------------------------------------------------------------------- +# Result normalization & comparison +# --------------------------------------------------------------------------- + +def _normalize_value(v: Any) -> Any: + if isinstance(v, float): + return round(v, 8) + if isinstance(v, int): + return float(v) + if isinstance(v, (list, tuple)): + return tuple(_normalize_value(x) for x in v) + if isinstance(v, dict): + return {k: _normalize_value(v) for k, v in v.items()} + if v is None: + return None + if isinstance(v, bool): + return v + return str(v) + + +def _results_match(baseline_rows: list[tuple], candidate_rows: list[tuple]) -> bool: + """Compare two result sets. Handle ORDERED vs unordered comparison.""" + if len(baseline_rows) != len(candidate_rows): + return False + # Use normalized comparison + bn = [_normalize_value(r) for r in baseline_rows] + cn = [_normalize_value(r) for r in candidate_rows] + # Try ordered comparison first, fall back to sorted + if bn == cn: + return True + return sorted(bn) == sorted(cn) + + +# --------------------------------------------------------------------------- +# Timing +# --------------------------------------------------------------------------- + +def _time_query(conn: Any, sql: str, repeats: int = 3, timeout_s: int = 60) -> float: + """Execute a query `repeats` times and return the median time in seconds.""" + times = [] + for _ in range(repeats): + start = time.time() + try: + _run_sql(conn, sql, timeout_s) + except Exception as e: + raise RuntimeError(f"Query failed: {e}") from e + elapsed = time.time() - start + times.append(elapsed) + times.sort() + return times[len(times) // 2] + + +# --------------------------------------------------------------------------- +# Validation +# --------------------------------------------------------------------------- + +def _validate_index_def( + idx: dict[str, Any], conn: Any, +) -> tuple[bool, str]: + table = idx.get("table", "") + columns = idx.get("columns", []) + method = idx.get("method", "") + + if not table: + return False, "missing table name" + if not columns or not isinstance(columns, list): + return False, "columns must be a non-empty list" + if method != "btree": + return False, f"unsupported index method: {method}" + + if not _table_exists(conn, table): + return False, f"table does not exist: {table}" + + for col in columns: + if not _column_exists(conn, table, col): + return False, f"column does not exist: {table}.{col}" + + return True, "" + + +# --------------------------------------------------------------------------- +# Submission runner +# --------------------------------------------------------------------------- + +def _run_submission(script_path: Path, input_path: Path, output_path: Path, cwd: Path) -> tuple[dict[str, Any], str, str]: + """Run a submission script and return parsed output.""" + result = subprocess.run( + [sys.executable, str(script_path), "--input", str(input_path), "--output", str(output_path)], + capture_output=True, text=True, timeout=60, cwd=str(cwd), + ) + if result.returncode != 0: + raise RuntimeError(f"Script failed (exit {result.returncode}): {result.stderr[:500]}") + + if not output_path.exists(): + raise RuntimeError(f"Output file not created: {output_path}") + + try: + submission = load_json(output_path) + except Exception as e: + raise RuntimeError(f"Invalid output JSON: {e}") from e + + return submission, result.stdout, result.stderr + + +# --------------------------------------------------------------------------- +# Score computation +# --------------------------------------------------------------------------- + +def _compute_metrics( + no_index_time: float, + baseline_time: float, + candidate_time: float, + n_indexes: int, + storage_mb: float, + data_size_mb: float, + correct: bool, + warmup_ok: bool, +) -> dict[str, Any]: + """Compute all metrics and validation flags.""" + + metrics: dict[str, Any] = { + "valid": 0.0, + "combined_score": INVALID_COMBINED_SCORE, + "no_index_time_s": round(no_index_time, 4), + "baseline_time_s": round(baseline_time, 4), + "candidate_time_s": round(candidate_time, 4), + "n_indexes": n_indexes, + "storage_mb": round(storage_mb, 2), + "data_size_mb": round(data_size_mb, 2), + "correct": 1.0 if correct else 0.0, + } + + # Hard constraints + valid = ( + correct + and warmup_ok + and n_indexes <= 10 + and storage_mb <= 500.0 + ) + + if not valid: + metrics["valid"] = 0.0 + metrics["combined_score"] = INVALID_COMBINED_SCORE + return metrics + + speedup = baseline_time / max(candidate_time, 1e-9) + metrics["speedup"] = round(speedup, 4) + + log_speedup = math.log2(max(speedup, 1e-6)) + storage_penalty = 0.3 * (storage_mb / 500.0) + count_penalty = 0.1 * (n_indexes / 10.0) + combined = log_speedup * (1.0 - storage_penalty - count_penalty) + + metrics["valid"] = 1.0 + metrics["combined_score"] = round(combined, 6) + metrics["log_speedup"] = round(log_speedup, 6) + metrics["storage_penalty"] = round(storage_penalty, 6) + metrics["count_penalty"] = round(count_penalty, 6) + + return metrics + + +# --------------------------------------------------------------------------- +# Main evaluation +# --------------------------------------------------------------------------- + +def evaluate( + program_path: str, + *, + timeout_s: float = 300.0, + repo_root: Path | None = None, +) -> Any: + """ + UnifiedTask evaluator interface. + + 1. Load config and generate submissions + 2. Start PostgreSQL, restore data + 3. Measure no-index, heuristic, and candidate performance + 4. Verify correctness + 5. Compute score + """ + artifacts: dict[str, Any] = {} + start_time = time.time() + + # Resolve paths + if repo_root is None: + repo_root = _find_repo_root() + task_dir = _task_dir(repo_root) + docker_image = "frontier-pg-index:latest" + raw_task_path = task_dir / "data" / "raw_task.json" + temp_dir = Path(tempfile.mkdtemp(prefix="fe_idxopt_")) + outputs_dir = task_dir / "outputs" + outputs_dir.mkdir(exist_ok=True) + + try: + # ------------------------------------------------------------------ + # Phase A: Generate submissions + # ------------------------------------------------------------------ + raw = load_json(raw_task_path) + queries = _resolve_queries(task_dir, raw) + schema = raw.get("schema", {}) + constraints = raw.get("constraints", {}) + artifacts["n_queries"] = len(queries) + artifacts["query_ids"] = [q.get("id", f"q{i}") for i, q in enumerate(queries)] + + # Run baseline (heuristic) + baseline_script = task_dir / "baseline" / "heuristic.py" + baseline_output = outputs_dir / "baseline_submission.json" + try: + baseline_sub, baseline_stdout, baseline_stderr = _run_submission( + baseline_script, raw_task_path, baseline_output, task_dir, + ) + artifacts["baseline_stdout"] = baseline_stdout[:2000] + except Exception as e: + artifacts["error_message"] = f"baseline failed: {e}" + return _wrap(_compute_metrics(0, 0, 0, 0, 0, 0, False, False), artifacts) + + # Run candidate + candidate_script = Path(program_path).resolve() + candidate_output = outputs_dir / "candidate_submission.json" + try: + candidate_sub, candidate_stdout, _ = _run_submission( + candidate_script, raw_task_path, candidate_output, task_dir, + ) + artifacts["candidate_stdout"] = candidate_stdout[:2000] + except Exception as e: + artifacts["error_message"] = f"candidate failed: {e}" + return _wrap(_compute_metrics(0, 0, 0, 0, 0, 0, False, False), artifacts) + + candidate_indexes = candidate_sub.get("indexes", []) + artifacts["candidate_n_indexes"] = len(candidate_indexes) + artifacts["candidate_indexes"] = candidate_indexes + + # ------------------------------------------------------------------ + # Phase B: Database evaluation + # ------------------------------------------------------------------ + container_id = None + port = None + + try: + container_id, port = _start_postgres(docker_image) + artifacts["postgres_port"] = port + conn = _pg_connect(port) + + # Restore data from pre-generated dump + dump_path = task_dir / "data" / "tpch_sf1" / "tpch_sf1.dump" + if not dump_path.exists(): + raise RuntimeError("no tpch_sf1.dump found — run data/tpch_sf1/gen_data.sh first") + _restore_dump(container_id, dump_path) + conn = _pg_connect(port) + + # Verify existing indexes in database match metadata + actual_idxs = _get_actual_indexes(conn) + artifacts["actual_indexes"] = actual_idxs + if len(actual_idxs) == 0: + raise RuntimeError("No primary key indexes created - schema may not be loaded correctly") + + # Prepare query list for timing + query_sqls = [q.get("sql", "") for q in queries] + query_ids = [q.get("id", f"q{i}") for i, q in enumerate(queries)] + + # -------------------------------------------------------------- + # Measure no-index time (reference only) + # -------------------------------------------------------------- + no_index_times = [] + for sql in query_sqls: + t = _time_query(conn, sql, repeats=3) + no_index_times.append(t) + no_index_total = sum(no_index_times) + artifacts["no_index_query_times"] = no_index_times + + # -------------------------------------------------------------- + # Measure heuristic time (baseline) + # -------------------------------------------------------------- + baseline_indexes = baseline_sub.get("indexes", []) + for idx in baseline_indexes: + ok, msg = _validate_index_def(idx, conn) + if not ok: + raise RuntimeError(f"Invalid baseline index: {msg}") + _create_single_index(conn, idx["table"], idx["columns"], idx["method"]) + conn.commit() + + # Warmup + for sql in query_sqls: + try: + _run_sql(conn, sql, timeout_s=30) + except Exception: + pass + + baseline_times = [] + for sql in query_sqls: + t = _time_query(conn, sql, repeats=3) + baseline_times.append(t) + baseline_total = sum(baseline_times) + artifacts["baseline_query_times"] = baseline_times + artifacts["baseline_indexes"] = baseline_indexes + + # Store baseline results for correctness comparison + baseline_results = [] + for sql in query_sqls: + baseline_results.append(_run_sql(conn, sql)) + conn.close() + + # -------------------------------------------------------------- + # Measure candidate time + # -------------------------------------------------------------- + _stop_postgres(container_id) + container_id = None + + # Fresh instance for candidate + container_id2, port2 = _start_postgres(docker_image) + conn2 = _pg_connect(port2) + _restore_dump(container_id2, dump_path) + conn2 = _pg_connect(port2) + + # Validate and create candidate indexes + for idx in candidate_indexes: + ok, msg = _validate_index_def(idx, conn2) + if not ok: + raise RuntimeError(f"Invalid candidate index: {msg}") + _create_single_index(conn2, idx["table"], idx["columns"], idx["method"]) + conn2.commit() + + # Measure storage + storage_mb = _get_index_size_mb(conn2) + data_size_mb = _get_data_size_mb(conn2) + + # Warmup + for sql in query_sqls: + try: + _run_sql(conn2, sql, timeout_s=30) + except Exception: + pass + + candidate_times = [] + for sql in query_sqls: + t = _time_query(conn2, sql, repeats=3) + candidate_times.append(t) + candidate_total = sum(candidate_times) + artifacts["candidate_query_times"] = candidate_times + + # Correctness verification + candidate_results = [] + for sql in query_sqls: + candidate_results.append(_run_sql(conn2, sql)) + + all_correct = True + mismatches = [] + for i, (br, cr) in enumerate(zip(baseline_results, candidate_results)): + if not _results_match(br, cr): + all_correct = False + mismatches.append(query_ids[i]) + artifacts["mismatch_queries"] = mismatches + + conn2.close() + _stop_postgres(container_id2) + + # Compute final score + metrics = _compute_metrics( + no_index_total, + baseline_total, + candidate_total, + len(candidate_indexes), + storage_mb, + data_size_mb, + all_correct, + True, + ) + metrics["runtime_s"] = round(time.time() - start_time, 4) + return _wrap(metrics, artifacts) + + except Exception as e: + traceback.print_exc() + artifacts["error_message"] = str(e)[:500] + err_metrics = { + "valid": 0.0, + "combined_score": INVALID_COMBINED_SCORE, + "no_index_time_s": 0.0, + "baseline_time_s": 0.0, + "candidate_time_s": 0.0, + "n_indexes": 0, + "storage_mb": 0.0, + "data_size_mb": 0.0, + "correct": 0.0, + "timeout": 1.0 if "timeout" in str(e).lower() else 0.0, + "runtime_s": round(time.time() - start_time, 4), + } + return _wrap(err_metrics, artifacts) + finally: + if container_id is not None: + try: + _stop_postgres(container_id) + except Exception: + pass + + except Exception as e: + traceback.print_exc() + artifacts["error_message"] = str(e)[:500] + return _wrap( + {"valid": 0.0, "combined_score": INVALID_COMBINED_SCORE, "runtime_s": round(time.time() - start_time, 4)}, + artifacts, + ) + finally: + shutil.rmtree(temp_dir, ignore_errors=True) + + +def _wrap(metrics: dict[str, Any], artifacts: dict[str, Any]) -> Any: + try: + from openevolve.evaluation_result import EvaluationResult + return EvaluationResult(metrics=metrics, artifacts=artifacts) + except ImportError: + return metrics + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + +def main() -> int: + if len(sys.argv) == 2 and not sys.argv[1].startswith("--"): + result = evaluate(sys.argv[1]) + metrics = result.metrics if hasattr(result, "metrics") else result + print(json.dumps(metrics)) + return 0 if metrics.get("valid", 0) > 0 else 1 + + parser = argparse.ArgumentParser(description="IndexOptimization evaluator") + parser.add_argument("program", help="Path to candidate program") + parser.add_argument("--timeout", type=float, default=300.0) + parser.add_argument("--repo-root", default=None) + args = parser.parse_args() + + repo_root = Path(args.repo_root).resolve() if args.repo_root else None + result = evaluate(args.program, timeout_s=args.timeout, repo_root=repo_root) + metrics = result.metrics if hasattr(result, "metrics") else result + print(json.dumps(metrics)) + return 0 if metrics.get("valid", 0) > 0 else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt b/benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt new file mode 100644 index 00000000..83ec1256 --- /dev/null +++ b/benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt @@ -0,0 +1 @@ +psycopg2-binary>=2.9 diff --git a/frontier_eval/conf/task/database_index_optimization.yaml b/frontier_eval/conf/task/database_index_optimization.yaml new file mode 100644 index 00000000..9cf1abca --- /dev/null +++ b/frontier_eval/conf/task/database_index_optimization.yaml @@ -0,0 +1,5 @@ +name: unified +benchmark: ComputerSystems/IndexOptimization +runtime: + isolation_mode: docker + docker_image: frontier-pg-index:latest From a388f1eb8e9afb26c707f1da797292fb30ad623c Mon Sep 17 00:00:00 2001 From: han Date: Wed, 15 Jul 2026 12:13:15 +0800 Subject: [PATCH 3/3] feat(ComputerSystems): add CacheReplacement benchmark (replaces IndexOptimization) CacheReplacement: CPU cache replacement policy optimization via ChampSim. - Agent modifies C++ replacement policy (EVOLVE-BLOCK) - Score: GMEAN(IPC_candidate / IPC_LRU), baseline = 1.0 - Quick: 3 traces (~15 min) | Full: 10 traces (~30-60 min) - 10 validation tests pass, verified on Apple Silicon --- TASK_DETAILS.md | 6 +- .../CacheReplacement/.gitignore | 9 + .../CacheReplacement/README.md | 284 ++++++++ .../CacheReplacement/README_zh-CN.md | 202 ++++++ .../ComputerSystems/CacheReplacement/Task.md | 368 ++++++++++ .../CacheReplacement/Task_zh-CN.md | 139 ++++ .../CacheReplacement/baseline/lru.cc | 21 + .../CacheReplacement/baseline/random.cc | 11 + .../CacheReplacement/baseline/result_log.txt | 53 ++ .../CacheReplacement/data/README.md | 59 ++ .../CacheReplacement/data/checksums.txt | 14 + .../CacheReplacement/data/download_traces.sh | 8 + .../frontier_eval/agent_files.txt | 18 + .../frontier_eval/artifact_files.txt | 2 + .../frontier_eval/candidate_destination.txt | 1 + .../frontier_eval/constraints.txt | 8 + .../frontier_eval/copy_files.txt | 0 .../frontier_eval/eval_command.txt | 1 + .../frontier_eval/eval_cwd.txt | 0 .../frontier_eval/initial_program.txt | 1 + .../frontier_eval/readonly_files.txt | 14 + .../references/constants.json | 17 + .../references/problem_config.json | 17 + .../references/related_work.md | 61 ++ .../CacheReplacement/replacement/my_policy.cc | 29 + .../CacheReplacement/replacement/my_policy.h | 20 + .../CacheReplacement/verification/Dockerfile | 84 +++ .../verification/evaluator.py | 223 ++++++ .../verification/requirements.txt | 1 + .../verification/test_validation.py | 55 ++ .../verification/verify_trace.xz | Bin 0 -> 74756 bytes .../IndexOptimization/README.md | 64 -- .../IndexOptimization/README_zh-CN.md | 38 - .../ComputerSystems/IndexOptimization/Task.md | 83 --- .../IndexOptimization/Task_zh-CN.md | 53 -- .../IndexOptimization/baseline/heuristic.py | 130 ---- .../IndexOptimization/baseline/result_log.txt | 36 - .../IndexOptimization/data/raw_task.json | 61 -- .../data/tpch_sf1/gen_data.sh | 114 --- .../data/tpch_sf1/queries/tpch_q1.json | 10 - .../data/tpch_sf1/queries/tpch_q10.json | 19 - .../data/tpch_sf1/queries/tpch_q12.json | 19 - .../data/tpch_sf1/queries/tpch_q3.json | 17 - .../data/tpch_sf1/queries/tpch_q5.json | 22 - .../data/tpch_sf1/queries/tpch_q6.json | 15 - .../data/tpch_sf1/schema.sql | 95 --- .../frontier_eval/agent_files.txt | 10 - .../frontier_eval/artifact_files.txt | 2 - .../frontier_eval/candidate_destination.txt | 1 - .../frontier_eval/constraints.txt | 13 - .../frontier_eval/eval_command.txt | 1 - .../frontier_eval/initial_program.txt | 1 - .../frontier_eval/readonly_files.txt | 4 - .../references/constants.json | 23 - .../IndexOptimization/scripts/init.py | 92 --- .../verification/docker/Dockerfile | 7 - .../verification/evaluator.py | 672 ------------------ .../verification/requirements.txt | 1 - benchmarks/ComputerSystems/README.md | 1 + benchmarks/ComputerSystems/README_zh-CN.md | 1 + 60 files changed, 1727 insertions(+), 1604 deletions(-) create mode 100644 benchmarks/ComputerSystems/CacheReplacement/.gitignore create mode 100644 benchmarks/ComputerSystems/CacheReplacement/README.md create mode 100644 benchmarks/ComputerSystems/CacheReplacement/README_zh-CN.md create mode 100644 benchmarks/ComputerSystems/CacheReplacement/Task.md create mode 100644 benchmarks/ComputerSystems/CacheReplacement/Task_zh-CN.md create mode 100644 benchmarks/ComputerSystems/CacheReplacement/baseline/lru.cc create mode 100644 benchmarks/ComputerSystems/CacheReplacement/baseline/random.cc create mode 100644 benchmarks/ComputerSystems/CacheReplacement/baseline/result_log.txt create mode 100644 benchmarks/ComputerSystems/CacheReplacement/data/README.md create mode 100644 benchmarks/ComputerSystems/CacheReplacement/data/checksums.txt create mode 100755 benchmarks/ComputerSystems/CacheReplacement/data/download_traces.sh create mode 100644 benchmarks/ComputerSystems/CacheReplacement/frontier_eval/agent_files.txt create mode 100644 benchmarks/ComputerSystems/CacheReplacement/frontier_eval/artifact_files.txt create mode 100644 benchmarks/ComputerSystems/CacheReplacement/frontier_eval/candidate_destination.txt create mode 100644 benchmarks/ComputerSystems/CacheReplacement/frontier_eval/constraints.txt rename benchmarks/ComputerSystems/{IndexOptimization => CacheReplacement}/frontier_eval/copy_files.txt (100%) create mode 100644 benchmarks/ComputerSystems/CacheReplacement/frontier_eval/eval_command.txt rename benchmarks/ComputerSystems/{IndexOptimization => CacheReplacement}/frontier_eval/eval_cwd.txt (100%) create mode 100644 benchmarks/ComputerSystems/CacheReplacement/frontier_eval/initial_program.txt create mode 100644 benchmarks/ComputerSystems/CacheReplacement/frontier_eval/readonly_files.txt create mode 100644 benchmarks/ComputerSystems/CacheReplacement/references/constants.json create mode 100644 benchmarks/ComputerSystems/CacheReplacement/references/problem_config.json create mode 100644 benchmarks/ComputerSystems/CacheReplacement/references/related_work.md create mode 100644 benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.cc create mode 100644 benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.h create mode 100644 benchmarks/ComputerSystems/CacheReplacement/verification/Dockerfile create mode 100644 benchmarks/ComputerSystems/CacheReplacement/verification/evaluator.py create mode 100644 benchmarks/ComputerSystems/CacheReplacement/verification/requirements.txt create mode 100644 benchmarks/ComputerSystems/CacheReplacement/verification/test_validation.py create mode 100644 benchmarks/ComputerSystems/CacheReplacement/verification/verify_trace.xz delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/README.md delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/Task.md delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/references/constants.json delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/scripts/init.py delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py delete mode 100644 benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt diff --git a/TASK_DETAILS.md b/TASK_DETAILS.md index b331f7d8..c89789af 100644 --- a/TASK_DETAILS.md +++ b/TASK_DETAILS.md @@ -204,7 +204,7 @@ We welcome new engineering problem ideas — even without complete verification Polarization-multiplexed holography - ComputerSystems + ComputerSystems MallocLab High-performance C memory allocator (utilization & throughput) @@ -212,6 +212,10 @@ We welcome new engineering problem ideas — even without complete verification DuckDBWorkloadOptimization Index / materialized-view selection and query rewriting on official DuckDB workloads + + CacheReplacement + CPU cache replacement policy optimization via ChampSim on SPEC CPU traces + EngDesign CY_03, WJ_01, XY_05, AM_02, AM_03, YJ_02, YJ_03 diff --git a/benchmarks/ComputerSystems/CacheReplacement/.gitignore b/benchmarks/ComputerSystems/CacheReplacement/.gitignore new file mode 100644 index 00000000..4a14c86f --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/.gitignore @@ -0,0 +1,9 @@ +data/traces/ +data/ChampSim/ +data/baseline_ipcs/ +references/generated_baseline.json +__pycache__/ +*.pyc +*.log +temp/ +.DS_Store diff --git a/benchmarks/ComputerSystems/CacheReplacement/README.md b/benchmarks/ComputerSystems/CacheReplacement/README.md new file mode 100644 index 00000000..522a60d1 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/README.md @@ -0,0 +1,284 @@ +# CacheReplacement — CPU Cache Replacement Policy Optimization + +An AI agent designs a CPU cache replacement policy by modifying C++ code. The policy is compiled into the [ChampSim](https://github.com/ChampSim/ChampSim) simulator and evaluated on SPEC CPU 2017 memory access traces. The agent's goal is to outperform the classic LRU baseline. + +| Quick Links | | +|---|---| +| Task specification | [Task.md](Task.md) / [Task_zh-CN.md](Task_zh-CN.md) | +| Engineering domain | Computer Architecture — microarchitectural optimization | +| Existing benchmarks comparison | MallocLab (OS), DuckDB (Database) → **CacheReplacement (Architecture)** | +| Competition background | [Cache Replacement Championship (CRC-2, ISCA 2017)](https://crc2.ece.tamu.edu/) | + +--- + +## Architecture Pipeline + +``` + Agent edits replacement/my_policy.cc (within EVOLVE-BLOCK markers) + │ + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 1: Source Validation │ + │ · Best-effort static scan │ + │ · Forbidden headers: , , │ + │ · Forbidden APIs: fopen, mmap, system, fork │ + │ · No randomness: rand(), mt19937, time() │ + │ · No dynamic memory: new, delete, malloc, free │ + │ · Storage budget ≤ 64 KB (approximate check) │ + │ · EVOLVE-BLOCK integrity check │ + │ FAIL → valid=0, score=0 │ + └──────────────────────┬───────────────────────────────┘ + │ PASS + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 2: Compile into ChampSim │ + │ · config.sh + make -j4 (incremental build) │ + │ · Environment: g++-13, C++17, -O3 -fno-exceptions │ + │ · Timeout: 300s │ + │ FAIL → valid=0, score=0 │ + └──────────────────────┬───────────────────────────────┘ + │ PASS + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 3: Quick Verify │ + │ · Run on verify_trace.xz (10M warmup + 20M sim) │ + │ · Checks: no crash, valid IPC value │ + │ · Purpose: fast rejection of broken policies │ + │ · NOT used for scoring │ + │ · Timeout: 120s │ + │ FAIL → valid=0, score=0 │ + └──────────────────────┬───────────────────────────────┘ + │ PASS + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 4: Full Evaluation │ + │ · Quick mode: 3 traces, ~15 min │ + │ · Full mode: 10 traces, ~30-60 min │ + │ · Per-trace: 50M warmup + 100M simulation │ + │ · IPC extracted from ChampSim stdout │ + │ · score = GMEAN(IPC_candidate / IPC_LRU) │ + │ · Failed traces are skipped (remaining continue) │ + └──────────────────────┬───────────────────────────────┘ + │ + ▼ + Output: {valid, score, per_trace, error} +``` + +--- + +## Scoring + +``` +score = geometric_mean( IPC_candidate[t] / IPC_LRU[t] ) over all completed traces t + + score = 1.0 → parity with LRU baseline + score > 1.0 → better than LRU + score ∈ (0,1) → worse than LRU but functionally valid + score = 0 → invalid (safety/compile/runtime failure) +``` + +Baseline LRU vs LRU always gives score = 1.0. Score > 1.0 means the agent discovered measurable improvement over a decades-old heuristic. + +### Metrics output (metrics.json) + +```json +{ + "valid": true, + "score": 1.15, + "benchmark_version": "1.0.0", + "mode": "full", + "compile_time_s": 7.8, + "simulation_time_s": 914.7, + "num_traces_completed": 10, + "error": null +} +``` + +### On failure + +```json +{ + "valid": false, + "score": 0.0, + "error": "Static validation: Forbidden header: " +} +``` + +--- + +## Evaluation Modes + +| Mode | Traces | Warmup | Simulation | Expected runtime | Use case | +|---|---|---|---|---|---| +| **Quick** (`--mode quick`) | 3 (mcf, x264, cactuBSSN) | 10M | 20M | ~15 min | Agent iteration | +| **Full** (`--mode full`) | All 10 | 50M | 100M | ~30-60 min | Final scoring | + +--- + +## File Structure + +``` +CacheReplacement/ +├── replacement/ +│ ├── my_policy.h # Class declaration (EVOLVE-BLOCK for member vars) +│ └── my_policy.cc # Method implementation (EVOLVE-BLOCK for methods) ← Agent target +├── verification/ +│ ├── evaluator.py # 4-layer evaluation pipeline +│ ├── test_validation.py # 10 Layer-1 validation tests +│ ├── Dockerfile # Ubuntu 22.04, gcc-13, cmake 3.22 +│ └── requirements.txt # pyyaml +├── baseline/ +│ ├── lru.cc # LRU reference (official baseline) +│ ├── random.cc # Random policy (sanity check only, score < 1.0) +│ ├── metrics.json # Local test results +│ └── result_log.txt # Test environment & results +├── frontier_eval/ # Frontier-Eng integration metadata +├── references/ +│ ├── constants.json # Parameter definitions +│ ├── problem_config.json # Trace lists, timeout config +│ ├── generated_baseline.json # Auto-generated LRU IPC cache +│ └── related_work.md # Academic references +└── data/ # Auto-downloaded (NOT in git) + ├── download_traces.sh # Automated download script + ├── checksums.txt # SHA256 checksums + ├── traces/ # 10 SPEC CPU 2017 .xz traces + └── ChampSim/ # Simulator source (commit 51588e1d) +``` + +--- + +## EVOLVE-BLOCK Contract Summary + +The agent modifies **two files**, each with `// EVOLVE-BLOCK-START` / `// EVOLVE-BLOCK-END` markers: + +### `replacement/my_policy.h` — Member variables (within EVOLVE-BLOCK) + +```cpp +// EVOLVE-BLOCK-START +// Agent may add fixed-size arrays here: +uint8_t history_table[1024]; // 1 KB +uint16_t access_counters[64]; // 128 bytes +// Total ≤ 64 KB (checked by evaluator) +// EVOLVE-BLOCK-END +``` + +### `replacement/my_policy.cc` — Method implementations (within EVOLVE-BLOCK) + +```cpp +// EVOLVE-BLOCK-START +// Agent may modify: +// - find_victim() body → eviction decision logic +// - replacement_cache_fill() → fill handler +// - update_replacement_state() → state update +// +// Agent must NOT modify: +// - function signatures +// - code outside EVOLVE-BLOCK +// EVOLVE-BLOCK-END +``` + +### Forbidden (causes immediate rejection with valid=0) + +| Category | Examples | Detection | +|---|---|---| +| **Headers** | ``, ``, ``, ``, ``, ``, `` | Header scan | +| **File I/O** | `fopen`, `fread`, `fwrite`, `open(`, `read(`, `write(`, `mmap(`, `munmap(` | API scan | +| **System** | `system(`, `popen`, `fork` | API scan | +| **Threading** | `pthread_`, `std::thread`, `std::async` | API scan | +| **Dynamic memory** | `new`, `delete`, `malloc(`, `free(`, `calloc(`, `realloc(` | Regex scan | +| **Dynamic containers** | `std::vector`, `std::map`, `std::unordered_map`, `std::set`, `std::list`, `std::deque` | Container scan | +| **Non-determinism** | `rand()`, `srand(`, `random_device`, `mt19937`, `time(`, `clock(` | Function scan | +| **Exceptions/RTTI** | `throw`, `try`, `catch`, `dynamic_cast`, `typeid` | Keyword scan | +| **Storage overflow** | Total declared arrays > 64 KB | Static size sum | +| **Interface violation** | Missing EVOLVE-BLOCK markers, modified read-only code | Diff check | + +--- + +## Setup + +### Prerequisites + +- **Python 3.10+** +- **C++17 capable compiler** (g++-12+ or clang) +- **CMake ≥ 3.22** +- **~10 GB free disk space** (for trace download + ChampSim build) + +### Step-by-step + +```bash +# 1. Clone ChampSim simulator +git clone https://github.com/ChampSim/ChampSim.git data/ChampSim +cd data/ChampSim +git checkout 51588e1d6f97875fe8de1a3621d28668bff83fcf +git submodule update --init +./vcpkg/bootstrap-vcpkg.sh +./vcpkg/vcpkg install +cd ../.. + +# 2. Download SPEC CPU 2017 traces +bash data/download_traces.sh + +# 3. Install Python dependencies +pip install -r verification/requirements.txt + +# 4. Run baseline test (quick mode) +python verification/evaluator.py replacement/my_policy.cc --mode quick +``` + +### Docker + +```bash +docker build -t champsim-eval -f verification/Dockerfile . +docker run --rm -v $(pwd):/benchmark champsim-eval +``` + +--- + +## Validation Tests + +```bash +python verification/test_validation.py +``` + +Expected output: all 10 tests pass: + +``` + [PASS] 1. Normal LRU policy: clean + [PASS] 2. Forbidden header : Forbidden header: + [PASS] 3. rand() usage: Non-deterministic function: rand() + [PASS] 4. fopen() file I/O: Forbidden API: fopen + [PASS] 5. std::vector in EVOLVE-BLOCK: Dynamic container: std::vector + [PASS] 6. new operator: Dynamic memory: ... + [PASS] 7. mt19937 randomness: Non-deterministic function: mt19937 + [PASS] 8. system() call: Forbidden API: system( + [PASS] 9. Missing EVOLVE-BLOCK-END: Missing '// EVOLVE-BLOCK-END' in .cc file + [PASS] 10. Budget exceeded: Storage budget exceeded: ~100000 B > 65536 B +``` + +--- + +## Integration with Frontier-Eng + +```bash +python -m frontier_eval task=unified task.benchmark=ComputerSystems/CacheReplacement algorithm.iterations=0 +``` + +--- + +## Differences from existing ComputerSystems benchmarks + +| Benchmark | Engineering domain | Agent's task | Language | +|---|---|---|---| +| `MallocLab` | OS / Memory management | Implement malloc/free/realloc | C | +| `DuckDBWorkloadOptimization` | Database / Query optimization | Select indexes + rewrite SQL | Python | +| **`CacheReplacement`** | **Computer architecture** | **Design CPU cache eviction policy** | **C++** | + +--- + +## References + +- Cache Replacement Championship (CRC-2, ISCA 2017): https://crc2.ece.tamu.edu/ +- Data Prefetching Championship (DPC-3, ISCA 2019): https://dpc3.compas.cs.stonybrook.edu/ +- ChampSim simulator: https://github.com/ChampSim/ChampSim +- SPEC CPU 2017 traces: https://dpc3.compas.cs.stonybrook.edu/champsim-traces/speccpu/ +- Related work: [references/related_work.md](references/related_work.md) diff --git a/benchmarks/ComputerSystems/CacheReplacement/README_zh-CN.md b/benchmarks/ComputerSystems/CacheReplacement/README_zh-CN.md new file mode 100644 index 00000000..d83a23ec --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/README_zh-CN.md @@ -0,0 +1,202 @@ +# CacheReplacement — CPU 缓存替换策略优化 + +AI agent 通过修改 C++ 代码设计 CPU 缓存替换策略，编译到 [ChampSim](https://github.com/ChampSim/ChampSim) 模拟器中，在 SPEC CPU 2017 内存访问 trace 上进行评测。目标是超越经典的 LRU 基线。 + +| 快速链接 | | +|---|---| +| 任务说明 | [Task_zh-CN.md](Task_zh-CN.md) / [Task.md](Task.md) | +| 工程领域 | 计算机体系结构 — 微架构优化 | +| 对比已有 benchmark | MallocLab (OS)、DuckDB (数据库) → **CacheReplacement (体系结构)** | +| 竞赛背景 | [Cache Replacement Championship (CRC-2, ISCA 2017)](https://crc2.ece.tamu.edu/) | + +--- + +## 评测管线 + +``` + Agent 修改 replacement/my_policy.cc（EVOLVE-BLOCK 标记内） + │ + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 1: 源码验证 │ + │ · 最佳努力静态扫描 │ + │ · 禁止头文件：, , │ + │ · 禁止API：fopen, mmap, system, fork │ + │ · 禁止随机：rand(), mt19937, time() │ + │ · 禁止动态内存：new, delete, malloc, free │ + │ · 存储预算 ≤ 64 KB（近似检查） │ + │ · EVOLVE-BLOCK 完整性检查 │ + │ 失败 → valid=0, score=0 │ + └──────────────────────┬───────────────────────────────┘ + │ 通过 + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 2: 编译到 ChampSim │ + │ · config.sh + make -j4（增量编译） │ + │ · 环境：g++-13, C++17, -O3 -fno-exceptions │ + │ · 超时：300s │ + │ 失败 → valid=0, score=0 │ + └──────────────────────┬───────────────────────────────┘ + │ 通过 + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 3: 快速验证 │ + │ · 运行 verify_trace（10M warmup + 20M sim） │ + │ · 检查：不崩溃、IPC合法 │ + │ · 目的：快速拒绝无效策略，不参与评分 │ + │ · 超时：120s │ + │ 失败 → valid=0, score=0 │ + └──────────────────────┬───────────────────────────────┘ + │ 通过 + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Layer 4: 正式评测 │ + │ · Quick 模式：3 traces，~15 min │ + │ · Full 模式：10 traces，~30-60 min │ + │ · 每个 trace：50M warmup + 100M simulation │ + │ · 从 ChampSim 输出提取 IPC │ + │ · score = GMEAN(IPC_candidate / IPC_LRU) │ + │ · 失败的 trace 跳过，剩余继续 │ + └──────────────────────┬───────────────────────────────┘ + │ + ▼ + 输出：{valid, score, per_trace, error} +``` + +--- + +## 评分 + +``` +score = geometric_mean( IPC_candidate[t] / IPC_LRU[t] ) + + score = 1.0 → 等价于 LRU 基线 + score > 1.0 → 优于 LRU + score ∈ (0,1) → 差于 LRU 但功能有效 + score = 0 → 无效（安全/编译/运行时失败） +``` + +### metrics.json 输出示例 + +```json +{ + "valid": true, + "score": 1.15, + "benchmark_version": "1.0.0", + "mode": "full", + "num_traces_completed": 10 +} +``` + +失败时： + +```json +{ + "valid": false, + "score": 0.0, + "error": "Static validation: Forbidden header: " +} +``` + +--- + +## 评测模式 + +| 模式 | Traces | Warmup | Simulation | 预期时间 | 用途 | +|---|---|---|---|---|---| +| **Quick** (`--mode quick`) | 3 (mcf, x264, cactuBSSN) | 10M | 20M | ~15 min | Agent 迭代 | +| **Full** (`--mode full`) | 全部 10 个 | 50M | 100M | ~30-60 min | 最终评分 | + +--- + +## EVOLVE-BLOCK 契约 + +Agent 修改 **两个文件**，每个都有 `// EVOLVE-BLOCK-START` / `// EVOLVE-BLOCK-END` 标记： + +### `replacement/my_policy.h` — 成员变量 + +```cpp +// EVOLVE-BLOCK-START +uint8_t history_table[1024]; // 1 KB +uint16_t counters[64]; // 128 bytes +// 总计 ≤ 64 KB +// EVOLVE-BLOCK-END +``` + +### `replacement/my_policy.cc` — 方法实现 + +```cpp +// EVOLVE-BLOCK-START +// 可修改：find_victim()、replacement_cache_fill()、update_replacement_state() +// 不可修改：函数签名、EVOLVE-BLOCK 外的代码 +// EVOLVE-BLOCK-END +``` + +### 禁止行为（立即拒绝，valid=0） + +| 类别 | 示例 | 检测方式 | +|---|---|---| +| **头文件** | ``, ``, ``, ``, `` | 头文件扫描 | +| **文件 I/O** | `fopen`, `fread`, `mmap(`, `open(` | API 扫描 | +| **系统调用** | `system(`, `popen`, `fork` | API 扫描 | +| **线程** | `pthread_`, `std::thread`, `std::async` | API 扫描 | +| **动态内存** | `new`, `delete`, `malloc(`, `free(` | 正则扫描 | +| **动态容器** | `std::vector`, `std::map`, `std::unordered_map` | 容器扫描 | +| **非确定性** | `rand()`, `srand(`, `random_device`, `mt19937`, `time()` | 函数扫描 | +| **存储溢出** | 声明的数组总大小 > 64 KB | 静态求和 | +| **接口违规** | 删除 EVOLVE-BLOCK 标记、修改只读区域 | diff 检测 | + +--- + +## 安装与运行 + +### 前置条件 + +- Python 3.10+ +- 支持 C++17 的编译器（g++-12+ 或 clang） +- CMake ≥ 3.22 +- ~10 GB 磁盘空间 + +### 安装步骤 + +```bash +# 1. 克隆 ChampSim +git clone https://github.com/ChampSim/ChampSim.git data/ChampSim +cd data/ChampSim && git checkout 51588e1d && git submodule update --init && ./vcpkg/bootstrap-vcpkg.sh && ./vcpkg/vcpkg install && cd ../.. + +# 2. 下载 traces +bash data/download_traces.sh + +# 3. 安装 Python 依赖 +pip install -r verification/requirements.txt + +# 4. 运行基线测试 +python verification/evaluator.py replacement/my_policy.cc --mode quick +``` + +### 验证测试 + +```bash +python verification/test_validation.py +``` + +预期：全部 10 个测试通过。 + +--- + +## 与现有 ComputerSystems benchmark 对比 + +| Benchmark | 工程领域 | Agent 操作 | 语言 | +|---|---|---|---| +| `MallocLab` | 操作系统内存管理 | 实现 malloc/free/realloc | C | +| `DuckDBWorkloadOptimization` | 数据库查询优化 | 选择索引 + 改写 SQL | Python | +| **`CacheReplacement`** | **计算机体系结构** | **设计缓存替换策略** | **C++** | + +--- + +## 参考资料 + +- Cache Replacement Championship (CRC-2): https://crc2.ece.tamu.edu/ +- Data Prefetching Championship (DPC-3): https://dpc3.compas.cs.stonybrook.edu/ +- ChampSim: https://github.com/ChampSim/ChampSim +- SPEC CPU 2017 traces: https://dpc3.compas.cs.stonybrook.edu/champsim-traces/speccpu/ diff --git a/benchmarks/ComputerSystems/CacheReplacement/Task.md b/benchmarks/ComputerSystems/CacheReplacement/Task.md new file mode 100644 index 00000000..2f74963e --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/Task.md @@ -0,0 +1,368 @@ +# CacheReplacement Task Specification + +## 1. Background + +### 1.1 The Engineering Problem + +CPU caches are finite-capacity hardware structures. On every cache miss, the hardware must decide which existing line to evict to make room for the incoming line. This eviction decision — the **replacement policy** — directly determines cache hit rate and application performance. + +- A 1% increase in LLC hit rate translates to 5–15% application speedup on modern CPUs +- Wrong eviction decisions cause cache pollution, thrashing, and severe performance degradation +- Engineers at Intel, AMD, ARM, and Apple have designed replacement policies for decades +- The problem remains open — no single policy dominates all workloads + +### 1.2 Why LRU is the Baseline + +LRU (Least Recently Used) is the classical replacement policy, used as the default baseline in virtually all replacement research (Cache Replacement Championship, Data Prefetching Championship, academic papers). It assumes temporal locality: recently accessed lines are likely to be accessed again soon. + +**Returning LRU does not mean optimization is successful.** Score = 1.0 means parity; score > 1.0 means the agent discovered measurable improvement over a decades-old heuristic. + +### 1.3 Competition Background + +- **Cache Replacement Championship (CRC-2)**, ISCA 2017: https://crc2.ece.tamu.edu/ +- **Data Prefetching Championship (DPC-3)**, ISCA 2019: https://dpc3.compas.cs.stonybrook.edu/ +- Winners: SHIP (signature-based hit prediction), Hawkeye (optimal reference simulation) + +### 1.4 Optimization Search Space + +| Dimension | What the agent designs | +|---|---| +| **Metadata structure** | What per-line state to track (counters, signatures, reuse distance, history) | +| **Recency / frequency** | How to balance temporal vs access-frequency signals | +| **Victim selection** | Decision logic in `find_victim()` given current state | +| **Adaptivity** | Whether to detect workload phases and switch strategies | +| **State compression** | How to pack metadata within 64 KB budget | + +--- + +## 2. Task Definition + +The agent modifies `replacement/my_policy.cc` (within `// EVOLVE-BLOCK-START` / `// EVOLVE-BLOCK-END` markers) to implement a custom cache replacement policy. The evalutor performs 4 layers of validation and evaluation: + +| Layer | Check | Outcome on failure | +|---|---|---| +| **L1: Source Validation** | Forbidden headers/APIs, randomness, dynamic memory, storage budget, EVOLVE-BLOCK integrity | `valid=0, score=0` | +| **L2: Compile** | Incremental build via ChampSim build system, timeout 300s | `valid=0, score=0` | +| **L3: Quick Verify** | Run on verify_trace.xz, check for crashes, timeout 120s | `valid=0, score=0` | +| **L4: Full Evaluation** | Run on 3 (quick) or 10 (full) SPEC traces, compute GMEAN score | Failed traces skipped | + +### 2.1 Input + +- `replacement/my_policy.h` — Class declaration (EVOLVE-BLOCK for member variables, read-only otherwise) +- `replacement/my_policy.cc` — Method implementations (EVOLVE-BLOCK for method bodies, read-only otherwise) + +### 2.2 Output + +The evaluator produces `metrics.json`: + +```json +{ + "valid": true, + "score": 1.15, + "benchmark_version": "1.0.0", + "mode": "full", + "compile_time_s": 7.8, + "simulation_time_s": 914.7, + "num_traces_completed": 10, + "num_traces_total": 10, + "error": null +} +``` + +On failure: + +```json +{ + "valid": false, + "score": 0.0, + "error": "Static validation: Forbidden header: " +} +``` + +--- + +## 3. EVOLVE-BLOCK Contract + +### 3.1 `replacement/my_policy.h` — Member Variable Declaration + +The EVOLVE-BLOCK in the header file allows the agent to **add** member variables that store replacement policy metadata: + +```cpp +// EVOLVE-BLOCK-START +// Agent may ADD member variables here (fixed-size arrays only): +// uint8_t name[SIZE]; +// std::array name; +// +// Agent must NOT remove or modify existing base members: +// long NUM_WAY, std::vector last_used_cycles, uint64_t cycle +// +// Total EVOLVE-BLOCK storage ≤ 64 KB (checked by evaluator). +// EVOLVE-BLOCK-END +``` + +**Allowed**: fixed-size primitive arrays, `std::array` +**Prohibited**: `std::vector`, `std::map`, dynamic containers, global/static variables + +### 3.2 `replacement/my_policy.cc` — Method Implementation + +The EVOLVE-BLOCK in the implementation file allows the agent to modify **three method bodies**: + +```cpp +// EVOLVE-BLOCK-START +// Agent may modify method bodies below. Keep function signatures unchanged. + +long my_policy::find_victim(uint32_t triggering_cpu, uint64_t instr_id, + long set, const champsim::cache_block* current_set, + champsim::address ip, champsim::address full_addr, + access_type type) +{ + // Return way index to evict: [0, NUM_WAY) + // Default LRU: evict way with oldest last-use cycle + auto begin = std::next(std::begin(last_used_cycles), set * NUM_WAY); + auto end = std::next(begin, NUM_WAY); + auto victim = std::min_element(begin, end); + return static_cast(std::distance(begin, victim)); +} + +void my_policy::replacement_cache_fill(uint32_t triggering_cpu, long set, + long way, champsim::address full_addr, + champsim::address ip, + champsim::address victim_addr, + access_type type) +{ + // Called when a new line is filled into the cache + last_used_cycles.at(static_cast(set * NUM_WAY + way)) = cycle++; +} + +void my_policy::update_replacement_state(uint32_t triggering_cpu, long set, + long way, champsim::address full_addr, + champsim::address ip, + champsim::address victim_addr, + access_type type, uint8_t hit) +{ + // Called on every cache access. Update policy state. + if (hit && access_type{type} != access_type::WRITE) + last_used_cycles.at(static_cast(set * NUM_WAY + way)) = cycle++; +} + +// EVOLVE-BLOCK-END +``` + +### 3.3 Available Base Members + +The following member variables are available for use in method implementations: + +| Member | Type | Description | +|---|---|---| +| `NUM_WAY` | `long` | Cache associativity (number of ways per set) | +| `last_used_cycles` | `std::vector` | Per-line last-use cycle timestamps (size = sets × ways) | +| `cycle` | `uint64_t` | Monotonically increasing cycle counter | + +### 3.4 Forbidden (complete list) + +| Category | Banned items | +|---|---| +| **Headers** | ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, ``, `` | +| **File I/O** | `fopen`, `fread`, `fwrite`, `fclose`, `open(`, `read(`, `write(`, `close(`, `mmap(`, `munmap(` | +| **System** | `system(`, `popen`, `fork` | +| **Threading** | `pthread_*`, `std::thread`, `std::async` | +| **Dynamic memory** | `new`, `delete`, `malloc(`, `free(`, `calloc(`, `realloc(` | +| **Dynamic containers** | `std::vector`, `std::map`, `std::unordered_map`, `std::set`, `std::list`, `std::deque` | +| **Non-determinism** | `rand()`, `srand(`, `random_device`, `mt19937`, `default_random_engine`, `time(`, `clock(` | +| **Exceptions** | `throw`, `try`, `catch` | +| **RTTI** | `dynamic_cast`, `typeid` | +| **Storage overflow** | Total declared arrays in header EVOLVE-BLOCK > 64 KB | +| **Interface violation** | Modifying function signatures, EVOLVE-BLOCK markers, or code outside EVOLVE-BLOCK | + +--- + +## 4. Scoring + +### 4.1 Formula + +``` +For each completed trace t: + ratio[t] = IPC_candidate[t] / IPC_LRU[t] + +final_score = exp( (1/N) * Σ ln(max(ratio[t], 0.001)) ) + where N = number of successfully completed traces +``` + +### 4.2 Interpretation + +| Score | Meaning | +|---|---| +| 1.0 | Parity with LRU (baseline) | +| > 1.0 | Better than LRU | +| (0, 1.0) | Worse than LRU but functionally valid | +| 0 | Invalid (any layer failure) | + +### 4.3 Edge Cases + +| Scenario | Handling | +|---|---| +| Candidate IPC ≤ 0 | Trace skipped | +| Simulation crashes | Trace skipped | +| Per-trace timeout | Trace skipped | +| All traces skipped | `valid=0, score=0` | +| Baseline IPC ≤ 0 | `valid=0, error="Bad environment"` | + +--- + +## 5. Evaluation Modes + +### Quick Mode (`--mode quick`) + +| Parameter | Value | +|---|---| +| Traces | `605.mcf_s-665B`, `625.x264_s-18B`, `607.cactuBSSN_s-2421B` | +| Warmup instructions | 10,000,000 | +| Simulation instructions | 20,000,000 | +| Expected runtime | ~15 minutes | +| Per-trace timeout | 600 seconds | +| Purpose | Agent iteration / rapid feedback | + +### Full Mode (`--mode full`) + +| Parameter | Value | +|---|---| +| Traces | 10 (all SPEC CPU 2017) | +| Warmup instructions | 50,000,000 | +| Simulation instructions | 100,000,000 | +| Expected runtime | 30–60 minutes | +| Per-trace timeout | 1800 seconds | +| Purpose | Leaderboard / final scoring | + +--- + +## 6. Timeout Behavior + +| Timeout type | Value | Effect | +|---|---|---| +| Compile | 300s | `valid=0, score=0` | +| Quick Verify | 120s | `valid=0, score=0` | +| Per-trace (Quick) | 600s | Trace skipped, remaining continue | +| Per-trace (Full) | 1800s | Trace skipped, remaining continue | + +--- + +## 7. Baselines + +### Official baseline: `baseline/lru.cc` + +LRU (Least Recently Used). Score = 1.0. Used for computing IPC normalization. + +### Sanity check: `baseline/random.cc` + +Random replacement policy. Expected score < 1.0. **Not an official baseline.** Not included in leaderboard. Used to verify that the evaluator can distinguish policy quality: + +```bash +# Expected: score < 1.0 (random is worse than LRU) +python verification/evaluator.py baseline/random.cc --mode quick +``` + +--- + +## 8. Examples + +### Example: Well-formed policy (LRU) + +This is the default implementation. It should produce `valid=1, score≈1.0`: + +```cpp +long my_policy::find_victim(...) { + auto begin = std::next(std::begin(last_used_cycles), set * NUM_WAY); + auto end = std::next(begin, NUM_WAY); + auto victim = std::min_element(begin, end); + return static_cast(std::distance(begin, victim)); +} +``` + +### Example: Rejected (iostream) + +```cpp +// EVOLVE-BLOCK-START +#include // ← Layer 1: Rejected. Forbidden header. +long my_policy::find_victim(...) { return 0; } +// EVOLVE-BLOCK-END +``` + +### Example: Rejected (rand) + +```cpp +// EVOLVE-BLOCK-START +long my_policy::find_victim(...) { + return rand() % NUM_WAY; // ← Layer 1: Rejected. Non-deterministic. +} +// EVOLVE-BLOCK-END +``` + +### Example: Rejected (deepseek vector) + +```cpp +// EVOLVE-BLOCK-START +std::vector extra; // ← Layer 1: Rejected. Dynamic container. +long my_policy::find_victim(...) { return 0; } +// EVOLVE-BLOCK-END +``` + +--- + +## 9. Usage + +```bash +# Quick mode (agent iteration, ~15 min) +python verification/evaluator.py replacement/my_policy.cc --mode quick + +# Full mode (final scoring, ~30-60 min) +python verification/evaluator.py replacement/my_policy.cc --mode full + +# Validation tests +python verification/test_validation.py + +# Frontier-Eng integration +python -m frontier_eval task=unified task.benchmark=ComputerSystems/CacheReplacement algorithm.iterations=0 +``` + +--- + +## 10. Data Management + +Traces are **ChampSim trace files generated from SPEC CPU 2017 workloads**, sourced from the [DPC-3 public repository](https://dpc3.compas.cs.stonybrook.edu/champsim-traces/speccpu/). + +> ⚠️ **License notice**: This benchmark repository does **not** redistribute SPEC CPU traces. The repository only contains metadata, download instructions, and checksums. Users must download trace files from the original source via `bash data/download_traces.sh`. + +### Trace specification + +| Trace | Domain | Quick | Full | Compressed | +|---|---|---|---|---| +| `603.bwaves_s-3699B` | Float (scientific) | — | ✓ | ~50 MB | +| `605.mcf_s-665B` | Integer (combinatorial) | ✓ | ✓ | ~60 MB | +| `625.x264_s-18B` | Integer (video) | ✓ | ✓ | ~80 MB | +| `654.roms_s-842B` | Float (ocean) | — | ✓ | ~55 MB | +| `657.xz_s-3167B` | Integer (compression) | — | ✓ | ~70 MB | +| `600.perlbench_s-210B` | Integer (scripting) | — | ✓ | ~60 MB | +| `602.gcc_s-734B` | Integer (compiler) | — | ✓ | ~75 MB | +| `607.cactuBSSN_s-2421B` | Float (physics) | ✓ | ✓ | ~90 MB | +| `621.wrf_s-575B` | Float (weather) | — | ✓ | ~65 MB | +| `631.deepsjeng_s-928B` | Integer (AI/game) | — | ✓ | ~55 MB | + +### Checksum verification + +```bash +cd data/traces && sha256sum -c ../checksums.txt +``` + +--- + +## 11. References + +- Cache Replacement Championship (CRC-2, ISCA 2017): https://crc2.ece.tamu.edu/ +- Data Prefetching Championship (DPC-3, ISCA 2019): https://dpc3.compas.cs.stonybrook.edu/ +- ChampSim simulator: https://github.com/ChampSim/ChampSim +- SPEC CPU 2017: https://www.spec.org/cpu2017/ +- SRRIP/DRRIP: Jaleel et al., ISCA 2010 +- SHIP: Wu et al., MICRO 2011 +- Hawkeye: Jain and Lin, ASPLOS 2016 +- TAGE-SC-L: Seznec, CBP 2016 +- See `references/related_work.md` for full citations. diff --git a/benchmarks/ComputerSystems/CacheReplacement/Task_zh-CN.md b/benchmarks/ComputerSystems/CacheReplacement/Task_zh-CN.md new file mode 100644 index 00000000..e7d93a1e --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/Task_zh-CN.md @@ -0,0 +1,139 @@ +# CacheReplacement 任务说明 + +## 1. 背景 + +### 1.1 工程问题 + +CPU 缓存容量有限。当新数据载入缓存时，必须选择驱逐哪条旧数据。这个决策——**替换策略（Replacement Policy）**——直接影响缓存命中率和性能。 + +- LLC 命中率提升 1% → 5–15% 应用加速 +- 错误驱逐会导致缓存污染和性能严重下降 +- Intel、AMD、ARM、Apple 研究数十年，至今无统一最优解 + +### 1.2 为什么 LRU 是基线 + +LRU 是经典替换策略，几乎所有学术研究以 LRU 为默认基线。 + +**返回 LRU 不代表优化成功。** Score = 1.0 = 等价；score > 1.0 = 超越了数十年启发式算法。 + +### 1.3 优化搜索空间 + +| 维度 | 设计内容 | +|---|---| +| metadata 结构 | 跟踪哪些状态（计数器、签名、重用距离） | +| 最近/频率 | 平衡时间局部性和频率 | +| 驱逐选择 | `find_victim()` 决策逻辑 | +| 自适应性 | 检测 workload 阶段切换 | +| 状态压缩 | 64 KB 预算内压缩 | + +--- + +## 2. 任务定义 + +修改 `replacement/my_policy.cc`（EVOLVE-BLOCK 内）实现替换策略。评估器 4 层验证： + +| 层 | 检查 | 失败结果 | +|---|---|---| +| L1: 源码验证 | 禁止头文件/API、动态内存、预算、标记完整性 | `valid=0, score=0` | +| L2: 编译 | ChampSim 增量编译，超时 300s | `valid=0, score=0` | +| L3: 快速验证 | verify_trace 运行不崩溃 | `valid=0, score=0` | +| L4: 正式评测 | 3/10 个 SPEC trace 计算 GMEAN | 失败 trace 跳过 | + +### 2.1 输入/输出 + +- 输入：`replacement/my_policy.h`（成员变量）、`replacement/my_policy.cc`（方法体） +- 输出：`metrics.json`（valid, score, per_trace） + +--- + +## 3. EVOLVE-BLOCK 契约 + +### 3.1 `.h` 文件 — 成员变量 + +```cpp +// EVOLVE-BLOCK-START +// 可添加固定大小数组，总 ≤ 64 KB +uint8_t history_table[1024]; +// EVOLVE-BLOCK-END +``` + +### 3.2 `.cc` 文件 — 方法体 + +可修改三个方法体，不可修改函数签名： + +- `find_victim()` — 返回 [0, NUM_WAY) +- `replacement_cache_fill()` — 缓存填充 +- `update_replacement_state()` — 状态更新 + +### 3.3 可用成员 + +`NUM_WAY`（long）、`last_used_cycles`（vector）、`cycle`（uint64_t） + +### 3.4 禁止项 + +| 类别 | 禁止项 | +|---|---| +| 头文件 | ``, ``, ``, ``, ``, `` | +| 文件 I/O | `fopen`, `fread`, `fwrite`, `open(`, `read(`, `mmap(` | +| 系统调用 | `system(`, `popen`, `fork` | +| 线程 | `pthread_`, `std::thread`, `std::async` | +| 动态内存 | `new`, `delete`, `malloc(`, `free(` | +| 动态容器 | `std::vector`, `std::map`, `std::unordered_map`, `std::set`, `std::list` | +| 非确定性 | `rand()`, `srand(`, `random_device`, `mt19937`, `time(` | +| 存储溢出 | 数组总大小 > 64 KB | +| 接口违规 | 修改函数签名、EVOLVE-BLOCK 标记、只读区域 | + +--- + +## 4. 评分 + +### 4.1 公式 + +``` +score = exp( (1/N) * Σ ln(max(IPC_candidate/ IPC_LRU, 0.001)) ) +``` + +### 4.2 解释 + +| 分数 | 含义 | +|---|---| +| 1.0 | LRU 等价（基线） | +| > 1.0 | 优于 LRU | +| (0, 1.0) | 差于但有效 | +| 0 | 无效 | + +--- + +## 5. 评测模式 + +| 模式 | Traces | Warmup | Sim | 时间 | 用途 | +|---|---|---|---|---|---| +| Quick | 3 | 10M | 20M | ~15 min | 迭代 | +| Full | 10 | 50M | 100M | ~30-60 min | 评分 | + +--- + +## 6. 使用方法 + +```bash +# Quick 模式 +python verification/evaluator.py replacement/my_policy.cc --mode quick + +# Full 模式 +python verification/evaluator.py replacement/my_policy.cc --mode full + +# 验证测试 +python verification/test_validation.py + +# Frontier-Eng 集成 +python -m frontier_eval task=unified task.benchmark=ComputerSystems/CacheReplacement algorithm.iterations=0 +``` + +--- + +## 7. 参考资料 + +- CRC-2: https://crc2.ece.tamu.edu/ +- DPC-3: https://dpc3.compas.cs.stonybrook.edu/ +- ChampSim: https://github.com/ChampSim/ChampSim +- 学术论文：`references/related_work.md` diff --git a/benchmarks/ComputerSystems/CacheReplacement/baseline/lru.cc b/benchmarks/ComputerSystems/CacheReplacement/baseline/lru.cc new file mode 100644 index 00000000..4e85ccf0 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/baseline/lru.cc @@ -0,0 +1,21 @@ +// baseline/lru.cc — LRU reference (official baseline) + +#include "cache.h" +#include "modules.h" + +struct lru : public champsim::modules::replacement { + long NUM_WAY; + std::vector last_used_cycles; + uint64_t cycle = 0; + lru(CACHE* cache) : lru(cache, cache->NUM_SET, cache->NUM_WAY) {} + lru(CACHE* cache, long sets, long ways) : replacement(cache), NUM_WAY(ways), last_used_cycles(static_cast(sets * ways), 0) {} + long find_victim(uint32_t, uint64_t, long set, const champsim::cache_block* b, champsim::address, champsim::address, access_type) { + return static_cast(std::distance(std::begin(last_used_cycles) + set * NUM_WAY, std::min_element(std::begin(last_used_cycles) + set * NUM_WAY, std::begin(last_used_cycles) + (set + 1) * NUM_WAY))); + } + void replacement_cache_fill(uint32_t, long set, long way, champsim::address, champsim::address, champsim::address, access_type) { + last_used_cycles.at(static_cast(set * NUM_WAY + way)) = cycle++; + } + void update_replacement_state(uint32_t, long set, long way, champsim::address, champsim::address, champsim::address, access_type, uint8_t hit) { + if (hit) last_used_cycles.at(static_cast(set * NUM_WAY + way)) = cycle++; + } +}; diff --git a/benchmarks/ComputerSystems/CacheReplacement/baseline/random.cc b/benchmarks/ComputerSystems/CacheReplacement/baseline/random.cc new file mode 100644 index 00000000..6df3093e --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/baseline/random.cc @@ -0,0 +1,11 @@ +// baseline/random.cc — sanity check only. Not official baseline. +#include "cache.h" +#include "modules.h" +struct random_policy : public champsim::modules::replacement { + long NUM_WAY; + random_policy(CACHE* cache) : random_policy(cache, cache->NUM_SET, cache->NUM_WAY) {} + random_policy(CACHE* cache, long sets, long ways) : replacement(cache), NUM_WAY(ways) {} + long find_victim(uint32_t, uint64_t, long, const champsim::cache_block*, champsim::address, champsim::address, access_type) { return rand() % NUM_WAY; } + void replacement_cache_fill(uint32_t, long, long, champsim::address, champsim::address, champsim::address, access_type) {} + void update_replacement_state(uint32_t, long, long, champsim::address, champsim::address, champsim::address, access_type, uint8_t) {} +}; diff --git a/benchmarks/ComputerSystems/CacheReplacement/baseline/result_log.txt b/benchmarks/ComputerSystems/CacheReplacement/baseline/result_log.txt new file mode 100644 index 00000000..85e78267 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/baseline/result_log.txt @@ -0,0 +1,53 @@ +# CacheReplacement Baseline Test Results +# Generated: 2026-07-15 (fresh run after all fixes) +# +# This file records the local test results of the LRU baseline +# policy (replacement/my_policy.cc default implementation). +# Score should be 1.0 (LRU vs LRU — baseline self-test). + +## Environment +- Host: macOS 14.5 (Apple Silicon M3) +- Python: 3.12.7 +- Compiler: Apple Clang 16 (C++17, via Xcode) +- ChampSim commit: 51588e1d6f97875fe8de1a3621d28668bff83fcf +- Evaluator: verification/evaluator.py v1.0.0 + +## Quick Mode Results +- Mode: quick (3 traces) +- Traces: 605.mcf_s-665B, 625.x264_s-18B, 607.cactuBSSN_s-2421B +- Warmup: 10M instructions | Simulation: 20M instructions +- Total runtime: 1882.3s +- Compile time: 7.1s +- Simulation time: 931.3s + +## Score +- Score: 1.0 (baseline LRU vs LRU = 1.0) +- Valid: true +- Traces completed: 3/3 + +## Per-Trace Results +See baseline/artifacts.json for complete per-trace IPC data. + +## Baseline LRU IPC (Quick Mode) +See references/generated_baseline.json for cached values. + +## Validation Tests +All 10 Layer-1 static validation tests pass: + 1. Normal LRU policy PASS + 2. Forbidden header PASS + 3. rand() usage PASS + 4. fopen() file I/O PASS + 5. Dynamic container PASS + 6. new operator PASS + 7. mt19937 randomness PASS + 8. system() call PASS + 9. Missing EVOLVE-BLOCK-END PASS + 10. Budget exceeded PASS + +## Reproducibility Notes +- ChampSim is pinned to commit 51588e1d via git checkout +- vcpkg dependencies managed via submodule (pinned) +- verify_trace.xz is a fixed file (75 KB, ~25s quick validation) +- Trace checksums are verified via data/checksums.txt (SHA256) +- Docker image uses pinned Ubuntu 22.04 + gcc-13 + cmake 3.22 +- On macOS, evaluator auto-detects vcpkg include path in _build_champsim() diff --git a/benchmarks/ComputerSystems/CacheReplacement/data/README.md b/benchmarks/ComputerSystems/CacheReplacement/data/README.md new file mode 100644 index 00000000..fc6203a6 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/data/README.md @@ -0,0 +1,59 @@ +# CacheReplacement — Data + +This directory contains external data required by the CacheReplacement benchmark. + +## Traces + +ChampSim trace files generated from SPEC CPU 2017 workloads. + +**Source**: [DPC-3 public repository](https://dpc3.compas.cs.stonybrook.edu/champsim-traces/speccpu/) + +> ⚠️ **License**: This benchmark does NOT redistribute SPEC CPU traces. +> Users must download traces from the original source and verify integrity. + +### Download + +```bash +# Automated download + checksum verification +bash data/download_traces.sh +``` + +### Verify + +```bash +# Verify all 10 traces match expected checksums +cd data/traces && sha256sum -c ../checksums.txt +``` + +### Trace list + +| Trace | Domain | Quick | Full | Size (xz) | +|---|---|---|---|---| +| `603.bwaves_s-3699B` | Float (scientific) | — | ✓ | ~50 MB | +| `605.mcf_s-665B` | Integer (combinatorial) | ✓ | ✓ | ~60 MB | +| `625.x264_s-18B` | Integer (video) | ✓ | ✓ | ~80 MB | +| `654.roms_s-842B` | Float (ocean) | — | ✓ | ~55 MB | +| `657.xz_s-3167B` | Integer (compression) | — | ✓ | ~70 MB | +| `600.perlbench_s-210B` | Integer (scripting) | — | ✓ | ~60 MB | +| `602.gcc_s-734B` | Integer (compiler) | — | ✓ | ~75 MB | +| `607.cactuBSSN_s-2421B` | Float (physics) | ✓ | ✓ | ~90 MB | +| `621.wrf_s-575B` | Float (weather) | — | ✓ | ~65 MB | +| `631.deepsjeng_s-928B` | Integer (AI/game) | — | ✓ | ~55 MB | + +**Total**: ~660 MB compressed, ~3 GB decompressed. + +## Simulator + +ChampSim is auto-cloned to `data/ChampSim/` at commit `51588e1d`. +Source: https://github.com/ChampSim/ChampSim + +## Structure + +``` +data/ +├── README.md ← this file +├── download_traces.sh ← automated download script +├── checksums.txt ← SHA256 checksums for all traces +├── traces/ ← .xz trace files (NOT tracked in git) +└── ChampSim/ ← simulator source (NOT tracked in git) +``` diff --git a/benchmarks/ComputerSystems/CacheReplacement/data/checksums.txt b/benchmarks/ComputerSystems/CacheReplacement/data/checksums.txt new file mode 100644 index 00000000..c8b72455 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/data/checksums.txt @@ -0,0 +1,14 @@ +# CacheReplacement — Trace SHA256 checksums +# Generated: 2026-07-15 +# Download: bash data/download_traces.sh +# Verify: cd data/traces && sha256sum -c ../checksums.txt +ac576730cff4f13d70384dff1939da83250bc0e3de1d9d994fdc637fd1b7cf4d 600.perlbench_s-210B.champsimtrace.xz +a32be73c53946748a1393cca4999f374c5d9d3737a414437d81bb20ed6311bd5 602.gcc_s-734B.champsimtrace.xz +107f5d325cb4f02a144657883176767fbe3153911e37e955201fff1fc6474594 603.bwaves_s-3699B.champsimtrace.xz +b7bef3f68180923fa4b88e1e8a6749487beef99099efce78500a03e3a3e8334e 605.mcf_s-665B.champsimtrace.xz +ae2cf28160028687bd3bdecf9135aeace9553b506d731230176087e1cee26559 607.cactuBSSN_s-2421B.champsimtrace.xz +03f8b049cfedf871715493513f505a726194fd09f10b0177f04113be80bead18 621.wrf_s-575B.champsimtrace.xz +e2782e14cff1fe6c2968f7527aaa4d5172eabc77a56e656cd6773c79696ee4ef 625.x264_s-18B.champsimtrace.xz +a4bf5970bf96d95e35007af9772989c4eaffb9312146f0bc6a13919270400e56 631.deepsjeng_s-928B.champsimtrace.xz +2377f1d782da5f4c158265d3a2379338903e15a56ce0aab013032539c181a82a 654.roms_s-842B.champsimtrace.xz +69b5c7c62231b4b116c6a49674e65af9d19ec3fd7bb6157c735d114c23e420ef 657.xz_s-3167B.champsimtrace.xz diff --git a/benchmarks/ComputerSystems/CacheReplacement/data/download_traces.sh b/benchmarks/ComputerSystems/CacheReplacement/data/download_traces.sh new file mode 100755 index 00000000..03a3d557 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/data/download_traces.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail +BASE_URL="https://dpc3.compas.cs.stonybrook.edu/champsim-traces/speccpu" +cd "$(dirname "$0")/traces" +for t in 603.bwaves_s-3699B 605.mcf_s-665B 625.x264_s-18B 654.roms_s-842B 657.xz_s-3167B 600.perlbench_s-210B 602.gcc_s-734B 607.cactuBSSN_s-2421B 621.wrf_s-575B 631.deepsjeng_s-928B; do + f="${t}.champsimtrace.xz"; [ -f "$f" ] && echo "Exists: $f" || { echo "Downloading $f..."; curl -sSL -o "$f" "${BASE_URL}/${f}" || wget -q -O "$f" "${BASE_URL}/${f}"; } +done +sha256sum -c ../checksums.txt 2>/dev/null && echo "Checksums OK" || echo "Checksums not verified" diff --git a/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/agent_files.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/agent_files.txt new file mode 100644 index 00000000..da6e439e --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/agent_files.txt @@ -0,0 +1,18 @@ +README.md +README_zh-CN.md +Task.md +Task_zh-CN.md +replacement/my_policy.cc +replacement/my_policy.h +verification/evaluator.py +verification/test_validation.py +baseline/lru.cc +baseline/random.cc +references/problem_config.json +references/constants.json +references/related_work.md +data/README.md +data/download_traces.sh +data/checksums.txt +frontier_eval/constraints.txt +.gitignore diff --git a/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/artifact_files.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/artifact_files.txt new file mode 100644 index 00000000..76dc893a --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/artifact_files.txt @@ -0,0 +1,2 @@ +metrics.json +artifacts.json diff --git a/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/candidate_destination.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/candidate_destination.txt new file mode 100644 index 00000000..443fd8c7 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/candidate_destination.txt @@ -0,0 +1 @@ +replacement/my_policy.cc diff --git a/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/constraints.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/constraints.txt new file mode 100644 index 00000000..c2ae9d5a --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/constraints.txt @@ -0,0 +1,8 @@ +CacheReplacement constraints: +1) Only modify code inside EVOLVE-BLOCK in replacement/my_policy.cc and replacement/my_policy.h. +2) Function signatures are read-only. +3) No dynamic memory, dynamic containers, file I/O, randomness, threading, system calls. +4) Storage budget: ≤64 KB for EVOLVE-BLOCK member variables. +5) Implementation must be deterministic. +6) Baseline: LRU. Score = geometric_mean(IPC_candidate / IPC_LRU). Baseline = 1.0. +7) Read-only: evaluator.py, baseline/*, references/*, data/ChampSim/* diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/copy_files.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/copy_files.txt similarity index 100% rename from benchmarks/ComputerSystems/IndexOptimization/frontier_eval/copy_files.txt rename to benchmarks/ComputerSystems/CacheReplacement/frontier_eval/copy_files.txt diff --git a/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/eval_command.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/eval_command.txt new file mode 100644 index 00000000..584b4178 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/eval_command.txt @@ -0,0 +1 @@ +{python} verification/evaluator.py {candidate} --metrics-out metrics.json --artifacts-out artifacts.json --mode full diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_cwd.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/eval_cwd.txt similarity index 100% rename from benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_cwd.txt rename to benchmarks/ComputerSystems/CacheReplacement/frontier_eval/eval_cwd.txt diff --git a/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/initial_program.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/initial_program.txt new file mode 100644 index 00000000..443fd8c7 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/initial_program.txt @@ -0,0 +1 @@ +replacement/my_policy.cc diff --git a/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/readonly_files.txt b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/readonly_files.txt new file mode 100644 index 00000000..3e9ff41a --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/frontier_eval/readonly_files.txt @@ -0,0 +1,14 @@ +replacement/my_policy.h +verification/evaluator.py +verification/test_validation.py +baseline/lru.cc +baseline/random.cc +references/problem_config.json +references/constants.json +references/related_work.md +data/README.md +data/download_traces.sh +frontier_eval/constraints.txt +frontier_eval/agent_files.txt +frontier_eval/artifact_files.txt +.gitignore diff --git a/benchmarks/ComputerSystems/CacheReplacement/references/constants.json b/benchmarks/ComputerSystems/CacheReplacement/references/constants.json new file mode 100644 index 00000000..57129114 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/references/constants.json @@ -0,0 +1,17 @@ +{ + "benchmark_version": "1.0.0", + "simulator": { + "name": "ChampSim", + "repo": "https://github.com/ChampSim/ChampSim", + "commit": "51588e1d6f97875fe8de1a3621d28668bff83fcf", + "build_command": "./config.sh .json && make -j4" + }, + "evaluation": { + "llc_size": "2 MB", + "llc_associativity": 16, + "cache_line_size": 64, + "num_cores": 1 + }, + "storage": { "budget_bytes": 65536, "validation": "approximate static check" }, + "scoring": { "formula": "GMEAN(IPC_candidate / IPC_LRU)", "baseline_score": 1.0 } +} diff --git a/benchmarks/ComputerSystems/CacheReplacement/references/problem_config.json b/benchmarks/ComputerSystems/CacheReplacement/references/problem_config.json new file mode 100644 index 00000000..50662e69 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/references/problem_config.json @@ -0,0 +1,17 @@ +{ + "benchmark_id": "cache_replacement", + "benchmark_version": "1.0.0", + "traces": { + "source": "https://dpc3.compas.cs.stonybrook.edu/champsim-traces/speccpu/", + "quick": ["605.mcf_s-665B", "625.x264_s-18B", "607.cactuBSSN_s-2421B"], + "full": [ + "603.bwaves_s-3699B", "605.mcf_s-665B", "625.x264_s-18B", + "654.roms_s-842B", "657.xz_s-3167B", "600.perlbench_s-210B", + "602.gcc_s-734B", "607.cactuBSSN_s-2421B", "621.wrf_s-575B", + "631.deepsjeng_s-928B" + ] + }, + "quick": { "warmup_instructions": 10000000, "sim_instructions": 20000000, "expected_runtime_min": "<10" }, + "full": { "warmup_instructions": 50000000, "sim_instructions": 100000000, "expected_runtime_min": "30-60" }, + "timeout": { "compile": 300, "quick_verify": 120, "per_trace_quick": 600, "per_trace_full": 1800 } +} diff --git a/benchmarks/ComputerSystems/CacheReplacement/references/related_work.md b/benchmarks/ComputerSystems/CacheReplacement/references/related_work.md new file mode 100644 index 00000000..1a1869b5 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/references/related_work.md @@ -0,0 +1,61 @@ +# Related Work — Cache Replacement Policies + +## Key Papers + +### LRU (Least Recently Used) + +Classic baseline replacement policy. Assumes temporal locality. Used as default baseline in virtually all replacement research. + +### SRRIP / DRRIP (Re-Reference Interval Prediction) + +Jaleel, A., Theobald, K. B., Steely, S. C., & Emer, J. +**"High Performance Cache Replacement Using Re-Reference Interval Prediction (RRIP)."** +*Proceedings of the 37th Annual International Symposium on Computer Architecture (ISCA)*, 2010. +DOI: [10.1145/1815961.1815971](https://doi.org/10.1145/1815961.1815971) + +### SHIP (Signature-based Hit Predictor) + +Wu, C.-J., Jaleel, A., Hasenplaugh, W., Martonosi, M., Steely, S. C., & Emer, J. +**"SHiP: Signature-based Hit Predictor for High Performance Caching."** +*Proceedings of the 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)*, 2011. +DOI: [10.1145/2155620.2155671](https://doi.org/10.1145/2155620.2155671) + +### Hawkeye + +Jain, A., & Lin, C. +**"Hawkeye: Efficiently Identifying Access Patterns for Optimal Cache Replacement."** +*Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)*, 2016. +DOI: [10.1145/2872362.2872406](https://doi.org/10.1145/2872362.2872406) + +### TAGE-SC-L + +Seznec, A. +**"Tage-SC-L Branch Predictors."** +*Championship Branch Prediction (CBP)*, 2016. + +### ChampSim Simulator + +Gober, N., Chacon, G., Wang, L., Gratz, P. V., Jimenez, D. A., Teran, E., Pugsley, S., & Kim, J. +**"The Championship Simulator: Architectural Simulation for Education and Competition."** +arXiv:2210.14324, 2022. +DOI: [10.48550/arXiv.2210.14324](https://doi.org/10.48550/arXiv.2210.14324) +GitHub: https://github.com/ChampSim/ChampSim + +## Competitions + +- **CRC-2** (Cache Replacement Championship 2, ISCA 2017) + https://crc2.ece.tamu.edu/ + +- **DPC-3** (Data Prefetching Championship 3, ISCA 2019) + https://dpc3.compas.cs.stonybrook.edu/ + +- **CBP-6** (Championship Branch Prediction 6, ISCA 2025) + https://ericrotenberg.wordpress.ncsu.edu/cbp2025-simulator-framework/ + +## Trace Sources + +- SPEC CPU 2017 traces for ChampSim (DPC-3 public mirror): + https://dpc3.compas.cs.stonybrook.edu/champsim-traces/speccpu/ + +- CRC-2 traces: + http://bit.ly/2t2nkUj diff --git a/benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.cc b/benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.cc new file mode 100644 index 00000000..0070cb9e --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.cc @@ -0,0 +1,29 @@ +// CacheReplacement — my_policy implementation +// Code outside EVOLVE-BLOCK is READ-ONLY. + +#include "my_policy.h" +#include +#include + +my_policy::my_policy(CACHE* cache) : my_policy(cache, cache->NUM_SET, cache->NUM_WAY) {} +my_policy::my_policy(CACHE* cache, long sets, long ways) : replacement(cache), NUM_WAY(ways), last_used_cycles(static_cast(sets * ways), 0) {} + +// EVOLVE-BLOCK-START +// Agent may modify method bodies below. Keep function signatures unchanged. + +long my_policy::find_victim(uint32_t, uint64_t, long set, const champsim::cache_block* b, champsim::address, champsim::address, access_type) { + auto begin = std::next(std::begin(last_used_cycles), set * NUM_WAY); + auto end = std::next(begin, NUM_WAY); + auto victim = std::min_element(begin, end); + return static_cast(std::distance(begin, victim)); +} + +void my_policy::replacement_cache_fill(uint32_t, long set, long way, champsim::address, champsim::address, champsim::address, access_type) { + last_used_cycles.at(static_cast(set * NUM_WAY + way)) = cycle++; +} + +void my_policy::update_replacement_state(uint32_t, long set, long way, champsim::address, champsim::address, champsim::address, access_type, uint8_t hit) { + if (hit) last_used_cycles.at(static_cast(set * NUM_WAY + way)) = cycle++; +} + +// EVOLVE-BLOCK-END diff --git a/benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.h b/benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.h new file mode 100644 index 00000000..bc58fa7c --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/replacement/my_policy.h @@ -0,0 +1,20 @@ +#ifndef REPLACEMENT_MY_POLICY_H +#define REPLACEMENT_MY_POLICY_H +#include +#include "cache.h" +#include "modules.h" +struct my_policy : public champsim::modules::replacement { + long NUM_WAY; + std::vector last_used_cycles; + uint64_t cycle = 0; + // EVOLVE-BLOCK-START + // Agent may ADD member variables here (fixed-size arrays only) + // Total EVOLVE-BLOCK storage ≤ 64 KB (checked by evaluator) + // EVOLVE-BLOCK-END + my_policy(CACHE* cache); + my_policy(CACHE* cache, long sets, long ways); + long find_victim(uint32_t, uint64_t, long, const champsim::cache_block*, champsim::address, champsim::address, access_type); + void replacement_cache_fill(uint32_t, long, long, champsim::address, champsim::address, champsim::address, access_type); + void update_replacement_state(uint32_t, long, long, champsim::address, champsim::address, champsim::address, access_type, uint8_t); +}; +#endif diff --git a/benchmarks/ComputerSystems/CacheReplacement/verification/Dockerfile b/benchmarks/ComputerSystems/CacheReplacement/verification/Dockerfile new file mode 100644 index 00000000..d1148e8b --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/verification/Dockerfile @@ -0,0 +1,84 @@ +FROM ubuntu:22.04 AS champsim-builder + +# Optional: use APT mirror for faster builds (e.g., ARG APT_MIRROR=mirrors.ustc.edu.cn) +ARG APT_MIRROR="archive.ubuntu.com" +# Optional: use GitHub mirror for cloning ChampSim (e.g., ARG GIT_BASE=https://gitclone.com/github.com/) +ARG GIT_BASE="https://github.com/" + +# Fixed toolchain (pinned versions for reproducibility) +RUN sed -i "s|archive.ubuntu.com|${APT_MIRROR}|g" /etc/apt/sources.list && \ + apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \ + build-essential \ + cmake=3.22.1-1ubuntu1 \ + g++-13=13.2.0-23ubuntu4 \ + gcc-13=13.2.0-23ubuntu4 \ + git=1:2.34.1-1ubuntu1.12 \ + wget=1.21.2-2ubuntu1 \ + xz-utils \ + python3=3.10.12-1~22.04 \ + python3-pip \ + && rm -rf /var/lib/apt/lists/* + +RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100 \ + && update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 100 + +# Clone ChampSim at fixed commit (supports GIT_BASE mirror) +WORKDIR /opt +RUN git clone ${GIT_BASE}ChampSim/ChampSim.git && \ + cd ChampSim && \ + git checkout 51588e1d6f97875fe8de1a3621d28668bff83fcf && \ + git submodule update --init && \ + ./vcpkg/bootstrap-vcpkg.sh && \ + ./vcpkg/vcpkg install + +# Env vars for ChampSim builds +ENV CXXFLAGS="-O3 -std=c++17 -fno-exceptions -fno-rtti" + +# Pre-build LRU baseline (caches build artifacts) +RUN cd /opt/ChampSim && \ + python3 config.sh config/my_policy_config.json && \ + mkdir -p config && \ + echo '{"executable_name":"champsim","block_size":64,"page_size":4096,"heartbeat_frequency":10000000,"num_cores":1,"ooo_cpu":[{"frequency":4000,"ifetch_buffer_size":64,"decode_buffer_size":32,"dispatch_buffer_size":32,"register_file_size":128,"rob_size":352,"lq_size":128,"sq_size":72,"fetch_width":6,"decode_width":6,"dispatch_width":6,"execute_width":4,"lq_width":2,"sq_width":2,"retire_width":5,"mispredict_penalty":1,"scheduler_size":128,"decode_latency":1,"dispatch_latency":1,"schedule_latency":0,"execute_latency":0,"branch_predictor":"bimodal","btb":"basic_btb"}],"L1I":{"sets":64,"ways":8,"rq_size":64,"wq_size":64,"pq_size":8,"mshr_size":8,"latency":4,"max_tag_check":2,"max_fill":2},"L1D":{"sets":64,"ways":12,"rq_size":64,"wq_size":64,"pq_size":8,"mshr_size":16,"latency":5,"max_tag_check":2,"max_fill":2,"prefetch_as_load":false,"virtual_prefetch":false,"replacement":"lru"},"L2C":{"sets":1024,"ways":8,"rq_size":32,"wq_size":32,"pq_size":16,"mshr_size":32,"latency":10,"max_tag_check":1,"max_fill":1,"prefetch_as_load":false,"virtual_prefetch":false,"replacement":"lru"},"LLC":{"sets":2048,"ways":16,"rq_size":32,"wq_size":32,"pq_size":32,"mshr_size":64,"latency":20,"max_tag_check":1,"max_fill":1,"prefetch_as_load":false,"virtual_prefetch":false,"replacement":"lru"}}' > config/lru_baseline.json && \ + python3 config.sh config/lru_baseline.json && \ + make -j4 + +# Build the my_policy module as well +RUN cd /opt/ChampSim && \ + echo '{"executable_name":"champsim","block_size":64,"page_size":4096,"heartbeat_frequency":10000000,"num_cores":1,"ooo_cpu":[{"frequency":4000,"ifetch_buffer_size":64,"decode_buffer_size":32,"dispatch_buffer_size":32,"register_file_size":128,"rob_size":352,"lq_size":128,"sq_size":72,"fetch_width":6,"decode_width":6,"dispatch_width":6,"execute_width":4,"lq_width":2,"sq_width":2,"retire_width":5,"mispredict_penalty":1,"scheduler_size":128,"decode_latency":1,"dispatch_latency":1,"schedule_latency":0,"execute_latency":0,"branch_predictor":"bimodal","btb":"basic_btb"}],"L1I":{"sets":64,"ways":8,"rq_size":64,"wq_size":64,"pq_size":8,"mshr_size":8,"latency":4,"max_tag_check":2,"max_fill":2},"L1D":{"sets":64,"ways":12,"rq_size":64,"wq_size":64,"pq_size":8,"mshr_size":16,"latency":5,"max_tag_check":2,"max_fill":2,"prefetch_as_load":false,"virtual_prefetch":false,"replacement":"my_policy"},"L2C":{"sets":1024,"ways":8,"rq_size":32,"wq_size":32,"pq_size":16,"mshr_size":32,"latency":10,"max_tag_check":1,"max_fill":1,"prefetch_as_load":false,"virtual_prefetch":false,"replacement":"my_policy"},"LLC":{"sets":2048,"ways":16,"rq_size":32,"wq_size":32,"pq_size":32,"mshr_size":64,"latency":20,"max_tag_check":1,"max_fill":1,"prefetch_as_load":false,"virtual_prefetch":false,"replacement":"my_policy"}}' > config/my_policy_config.json && \ + python3 config.sh config/my_policy_config.json && \ + make -j4 + +# Record environment versions +RUN echo "gcc: $(gcc --version | head -1)" > /opt/champsim.env && \ + echo "cmake: $(cmake --version | head -1)" >> /opt/champsim.env && \ + echo "python: $(python3 --version)" >> /opt/champsim.env && \ + echo "champsim: $(cd /opt/ChampSim && git rev-parse HEAD)" >> /opt/champsim.env + +# ── Evaluation stage ── +FROM ubuntu:22.04 +ARG APT_MIRROR="archive.ubuntu.com" +RUN sed -i "s|archive.ubuntu.com|${APT_MIRROR}|g" /etc/apt/sources.list && \ + apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \ + python3=3.10.12-1~22.04 \ + python3-pip \ + xz-utils \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=champsim-builder /opt/ChampSim /opt/ChampSim +COPY --from=champsim-builder /opt/champsim.env /opt/champsim.env + +RUN pip3 install --no-cache-dir pyyaml + +WORKDIR /benchmark +COPY . . + +# Symlink ChampSim +RUN ln -s /opt/ChampSim data/ChampSim + +# Download traces (continue on error) +RUN bash data/download_traces.sh 2>/dev/null; exit 0 + +# Generate baseline IPC +RUN python3 verification/evaluator.py replacement/my_policy.cc --mode quick --metrics-out /tmp/init_check.json 2>/dev/null; exit 0 + +CMD ["python3", "verification/evaluator.py", "replacement/my_policy.cc", "--mode", "full"] diff --git a/benchmarks/ComputerSystems/CacheReplacement/verification/evaluator.py b/benchmarks/ComputerSystems/CacheReplacement/verification/evaluator.py new file mode 100644 index 00000000..25fec24f --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/verification/evaluator.py @@ -0,0 +1,223 @@ +""" +CacheReplacement evaluator — 4-layer pipeline for ChampSim cache policy optimization. +""" + +import argparse, json, math, os, re, shutil, subprocess, sys, time, traceback +from pathlib import Path + +BENCHMARK_VERSION = "1.0.0" +CHAMPSIM_COMMIT = "51588e1d6f97875fe8de1a3621d28668bff83fcf" +STORAGE_BUDGET_BYTES = 65536 +TIMEOUT_COMPILE = 300; TIMEOUT_QUICK_VERIFY = 120 +TIMEOUT_PER_TRACE_QUICK = 600; TIMEOUT_PER_TRACE_FULL = 1800 +WARMUP_QUICK = 10_000_000; SIM_QUICK = 20_000_000 +WARMUP_FULL = 50_000_000; SIM_FULL = 100_000_000 + +FORBIDDEN_HEADERS = ["iostream","fstream","sstream","thread","mutex","future","shared_mutex","filesystem","unistd.h","fcntl.h","sys/stat.h","sys/mman.h","dlfcn.h","execinfo.h","signal.h","setjmp.h"] +FORBIDDEN_APIS = ["fopen","fread","fwrite","fclose","open(","read(","write(","close(","mmap(","munmap(","dlopen","dlsym","system(","popen","fork","pthread_","std::thread","std::async"] +FORBIDDEN_RANDOM = ["rand()","srand(","random_device","mt19937","default_random_engine","steady_clock","system_clock","time(","clock("] + +# Layer 1: Source Validation + +def _static_validation(cc_path: Path, h_path: Path | None) -> list[str]: + errors = [] + cc_text = cc_path.read_text() + if "// EVOLVE-BLOCK-START" not in cc_text: errors.append("Missing '// EVOLVE-BLOCK-START' in .cc file") + if "// EVOLVE-BLOCK-END" not in cc_text: errors.append("Missing '// EVOLVE-BLOCK-END' in .cc file") + s = cc_text.find("// EVOLVE-BLOCK-START"); e = cc_text.find("// EVOLVE-BLOCK-END") + if s >= 0 and e >= 0 and s >= e: errors.append("EVOLVE-BLOCK-START must precede EVOLVE-BLOCK-END") + for h in FORBIDDEN_HEADERS: + if f"#include <{h}>" in cc_text: errors.append(f"Forbidden header: <{h}>") + for api in FORBIDDEN_APIS: + if api in cc_text: errors.append(f"Forbidden API: {api}") + for rf in FORBIDDEN_RANDOM: + if rf in cc_text: errors.append(f"Non-deterministic function: {rf}") + dm_patterns = [r'\bnew\s+\w+', r'\bdelete\s+\[\]', r'\bdelete\s+\w+', r'\bmalloc\s*\(', r'\bfree\s*\(', r'\bcalloc\s*\(', r'\brealloc\s*\('] + for pat in dm_patterns: + if re.search(pat, cc_text): errors.append(f"Dynamic memory: {pat}") + for dc in ["std::vector","std::map","std::unordered_map","std::set","std::list","std::deque"]: + if dc in cc_text: errors.append(f"Dynamic container: {dc}") + if h_path and h_path.is_file(): + h_text = h_path.read_text() + hs = h_text.find("// EVOLVE-BLOCK-START"); he = h_text.find("// EVOLVE-BLOCK-END") + if hs >= 0 and he >= 0: + total = 0; type_sizes = {"uint8_t":1,"int8_t":1,"uint16_t":2,"int16_t":2,"uint32_t":4,"int32_t":4,"uint64_t":8,"int64_t":8,"bool":1,"char":1,"float":4,"double":8} + block = h_text[hs+len("// EVOLVE-BLOCK-START"):he] + for m in re.finditer(r'(uint\d+_t|int\d+_t|bool|char|float|double)\s+\w+\[(\d+)\]', block): + total += type_sizes.get(m.group(1),4) * int(m.group(2)) + for m in re.finditer(r'std::array\s*<\s*(\w+)\s*,\s*(\d+)\s*>', block): + total += type_sizes.get(m.group(1),4) * int(m.group(2)) + if total > STORAGE_BUDGET_BYTES: + errors.append(f"Storage budget exceeded: ~{total} B > {STORAGE_BUDGET_BYTES} B (approximate check)") + return errors + +# Layer 2: Compile (cross-platform) + +def _build_champsim(cd: Path, config_name: str = "my_policy_config") -> bool: + """Configure and build ChampSim. Handles vcpkg on macOS and Linux.""" + cfg = cd / "config" / f"{config_name}.json" + # Step 1: Run config.sh (generates Makefile + .csconfig) + p1 = subprocess.run([sys.executable, str(cd/"config.sh"), str(cfg)], cwd=cd, capture_output=True, text=True, timeout=60) + if p1.returncode != 0: return False + # Step 2: Ensure vcpkg include path is in absolute.options + # On Linux config.sh handles this; on macOS it may not. + abs_opts = cd / "absolute.options" + if abs_opts.is_file(): + vcpkg_incs = [str(p) for p in sorted((cd / "vcpkg_installed").glob("*/include"))] + for inc in vcpkg_incs: + if inc not in abs_opts.read_text(): + with open(abs_opts, "a") as f: + f.write(f" -isystem {inc}") + # Step 3: Build + p2 = subprocess.run(["make", "-j4"], cwd=cd, capture_output=True, text=True, timeout=TIMEOUT_COMPILE) + return p2.returncode == 0 + +def _ensure_config(cd: Path, name: str, replacement: str): + cfg = {"executable_name":"champsim","block_size":64,"page_size":4096,"heartbeat_frequency":10000000,"num_cores":1, + "ooo_cpu":[{"frequency":4000,"ifetch_buffer_size":64,"decode_buffer_size":32,"dispatch_buffer_size":32,"register_file_size":128,"rob_size":352,"lq_size":128,"sq_size":72,"fetch_width":6,"decode_width":6,"dispatch_width":6,"execute_width":4,"lq_width":2,"sq_width":2,"retire_width":5,"mispredict_penalty":1,"scheduler_size":128,"decode_latency":1,"dispatch_latency":1,"schedule_latency":0,"execute_latency":0,"branch_predictor":"bimodal","btb":"basic_btb"}], + "L1I":{"sets":64,"ways":8,"rq_size":64,"wq_size":64,"pq_size":8,"mshr_size":8,"latency":4,"max_tag_check":2,"max_fill":2}, + "L1D":{"sets":64,"ways":12,"rq_size":64,"wq_size":64,"pq_size":8,"mshr_size":16,"latency":5,"max_tag_check":2,"max_fill":2,"prefetch_as_load":False,"virtual_prefetch":False,"replacement":replacement}, + "L2C":{"sets":1024,"ways":8,"rq_size":32,"wq_size":32,"pq_size":16,"mshr_size":32,"latency":10,"max_tag_check":1,"max_fill":1,"prefetch_as_load":False,"virtual_prefetch":False,"replacement":replacement}, + "LLC":{"sets":2048,"ways":16,"rq_size":32,"wq_size":32,"pq_size":32,"mshr_size":64,"latency":20,"max_tag_check":1,"max_fill":1,"prefetch_as_load":False,"virtual_prefetch":False,"replacement":replacement}} + (cd/"config").mkdir(exist_ok=True); (cd/"config"/f"{name}.json").write_text(json.dumps(cfg,indent=2)) + +# Layer 3: Quick Verify (uses verify_trace.xz, ~2M instructions, ~25s) + +def _quick_verify(binary: Path, verify_trace: Path) -> tuple[bool, str]: + """Quick sanity check. NOT used for scoring.""" + cmd = [str(binary), "--warmup-instructions", "1000000", "--simulation-instructions", "2000000", str(verify_trace)] + p = subprocess.run(cmd, capture_output=True, text=True, timeout=TIMEOUT_QUICK_VERIFY) + if p.returncode != 0: return False, p.stderr[:500] + ipc = _parse_ipc(p.stdout) + return (True, p.stdout) if ipc and ipc > 0 else (False, f"Invalid IPC: {ipc}") + +# Layer 4: Full Evaluation + +def _run_trace(binary: Path, trace_path: Path, warmup: int, sim: int, timeout: int) -> dict: + cmd = [str(binary), "--warmup-instructions", str(warmup), "--simulation-instructions", str(sim), str(trace_path)] + try: + p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + if p.returncode != 0: return {"status":"failed","error":p.stderr[:200]} + ipc = _parse_ipc(p.stdout) + return ({"status":"ok","ipc":ipc} if ipc and ipc > 0 else {"status":"failed","error":f"invalid IPC: {ipc}"}) + except subprocess.TimeoutExpired: return {"status":"timeout"} + +def _parse_ipc(output: str) -> float | None: + for line in output.splitlines(): + m = re.search(r'cumulative IPC:\s+([\d.]+)', line) + if m: + try: return float(m.group(1)) + except ValueError: return None + return None + +def _get_baseline_ipcs(base_dir: Path, traces: list[str], warmup: int, sim: int) -> dict: + ref_dir = base_dir / "references"; cache_file = ref_dir / "generated_baseline.json" + if cache_file.is_file(): + data = json.loads(cache_file.read_text()) + if data.get("champsim_commit") == CHAMPSIM_COMMIT: return data.get("ipc_results", {}) + cd = base_dir / "data" / "ChampSim" + _ensure_config(cd, "lru_baseline", "lru") + if not _build_champsim(cd, "lru_baseline"): raise RuntimeError("Failed to build LRU baseline") + binary = cd / "bin" / "champsim"; td = base_dir / "data" / "traces" + to = TIMEOUT_PER_TRACE_FULL if sim == SIM_FULL else TIMEOUT_PER_TRACE_QUICK + results = {} + for t in traces: + tf = td / f"{t}.champsimtrace.xz" + if not tf.is_file(): continue + r = _run_trace(binary, tf, warmup, sim, to) + if r["status"] == "ok": results[t] = r["ipc"] + ref_dir.mkdir(parents=True, exist_ok=True) + cache_file.write_text(json.dumps({"benchmark_version":BENCHMARK_VERSION,"champsim_commit":CHAMPSIM_COMMIT,"warmup":warmup,"simulation":sim,"ipc_results":results},indent=2)) + return results + +# Main evaluate + +def evaluate(candidate_path: str, *, mode: str = "quick", repo_root: Path | None = None) -> dict: + start = time.time() + metrics = {"benchmark_version":BENCHMARK_VERSION,"simulator_commit":CHAMPSIM_COMMIT,"valid":0.0,"score":0.0,"runtime_s":0.0,"compile_time_s":0.0,"simulation_time_s":0.0,"l1_static_pass":0.0,"l2_compile_pass":0.0,"l3_quick_verify_pass":0.0,"l4_eval_pass":0.0,"num_traces_completed":0,"num_traces_total":0,"mode":mode,"error":None} + artifacts = {"per_trace_results":{}} + try: + repo = repo_root or _find_repo_root() + base = repo / "benchmarks" / "ComputerSystems" / "CacheReplacement" + cand = Path(candidate_path).expanduser().resolve() + artifacts["candidate_path"] = str(cand) + if mode == "quick": + traces = ["605.mcf_s-665B","625.x264_s-18B","607.cactuBSSN_s-2421B"] + warmup, sim, tto = WARMUP_QUICK, SIM_QUICK, TIMEOUT_PER_TRACE_QUICK + else: + traces = ["603.bwaves_s-3699B","605.mcf_s-665B","625.x264_s-18B","654.roms_s-842B","657.xz_s-3167B","600.perlbench_s-210B","602.gcc_s-734B","607.cactuBSSN_s-2421B","621.wrf_s-575B","631.deepsjeng_s-928B"] + warmup, sim, tto = WARMUP_FULL, SIM_FULL, TIMEOUT_PER_TRACE_FULL + if not cand.is_file(): metrics["error"] = "Candidate not found"; return _wrap(metrics, artifacts) + h_path = (cand.parent / f"{cand.stem}.h") if (cand.parent / f"{cand.stem}.h").is_file() else cand.with_suffix(".h") + se = _static_validation(cand, h_path if h_path.is_file() else None) + artifacts["static_errors"] = se + if se: metrics["error"] = f"Static validation: {'; '.join(se)}"; return _wrap(metrics, artifacts) + metrics["l1_static_pass"] = 1.0 + cd = base / "data" / "ChampSim"; md = cd / "replacement" / "my_policy" + md.mkdir(parents=True, exist_ok=True) + shutil.copy2(str(cand), str(md / "my_policy.cc")) + if h_path.is_file(): shutil.copy2(str(h_path), str(md / "my_policy.h")) + else: (md / "my_policy.h").write_text('#ifndef REPLACEMENT_MY_POLICY_H\n#define REPLACEMENT_MY_POLICY_H\n#include \n#include "cache.h"\n#include "modules.h"\nstruct my_policy;\n#endif\n') + t0 = time.perf_counter() + _ensure_config(cd, "my_policy_config", "my_policy") + if not _build_champsim(cd, "my_policy_config"): metrics["error"] = "Compilation failed"; return _wrap(metrics, artifacts) + metrics["compile_time_s"] = time.perf_counter() - t0; metrics["l2_compile_pass"] = 1.0 + binary = cd / "bin" / "champsim" + vt = base / "verification" / "verify_trace.xz" + if vt.is_file(): + qok, qlog = _quick_verify(binary, vt); artifacts["quick_verify_log"] = qlog + if not qok: metrics["error"] = "Quick verification failed"; return _wrap(metrics, artifacts) + metrics["l3_quick_verify_pass"] = 1.0 + baseline_ipcs = _get_baseline_ipcs(base, traces, warmup, sim) + if not baseline_ipcs: metrics["error"] = "Baseline generation failed"; return _wrap(metrics, artifacts) + td = base / "data" / "traces"; metrics["num_traces_total"] = float(len(traces)) + t1 = time.perf_counter(); results = {} + for trace_name in traces: + tf = td / f"{trace_name}.champsimtrace.xz" + results[trace_name] = _run_trace(binary, tf, warmup, sim, tto) if tf.is_file() else {"status":"failed","error":"trace not found"} + metrics["simulation_time_s"] = time.perf_counter() - t1; artifacts["per_trace_results"] = results + completed = {t:r for t,r in results.items() if r.get("status")=="ok"} + metrics["num_traces_completed"] = float(len(completed)) + if completed: + ratios = [] + for t,r in completed.items(): + bl = baseline_ipcs.get(t) + if bl and bl > 0 and r.get("ipc",0) > 0: ratios.append(r["ipc"]/bl) + if ratios: + metrics["score"] = math.exp(sum(math.log(max(x,0.001)) for x in ratios)/len(ratios)) + metrics["valid"] = 1.0; metrics["l4_eval_pass"] = 1.0 + else: metrics["error"] = "No valid trace results" + else: metrics["error"] = "No traces completed" + except subprocess.TimeoutExpired as exc: metrics["error"] = f"Timeout: {exc}" + except Exception as exc: metrics["error"] = str(exc); artifacts["traceback"] = traceback.format_exc() + finally: metrics["runtime_s"] = time.time() - start + return _wrap(metrics, artifacts) + +def _find_repo_root() -> Path: + here = Path(__file__).resolve() + for p in [here.parent, *here.parents]: + if (p / "benchmarks").is_dir() and (p / "frontier_eval").is_dir(): return p + return Path.cwd() + +def _wrap(m: dict, a: dict) -> dict: + try: + from openevolve.evaluation_result import EvaluationResult as ER + return ER(metrics=m, artifacts=a) + except ImportError: return {"metrics": m, "artifacts": a} + +def main() -> int: + p = argparse.ArgumentParser(description="CacheReplacement: evaluate a cache replacement policy via ChampSim simulation.") + p.add_argument("candidate", help="Path to the candidate my_policy.cc file (e.g., replacement/my_policy.cc)") + p.add_argument("--mode", choices=["quick", "full"], default="quick", + help="'quick' (3 traces, ~15 min) or 'full' (10 traces, ~30-60 min)") + p.add_argument("--metrics-out", default="", help="Optional path to write metrics JSON") + p.add_argument("--artifacts-out", default="", help="Optional path to write artifacts JSON") + args = p.parse_args() + result = evaluate(args.candidate, mode=args.mode) + r = result if isinstance(result,dict) else {"metrics":dict(result.metrics or {}),"artifacts":dict(result.artifacts or {})} + if args.metrics_out: Path(args.metrics_out).write_text(json.dumps(r["metrics"],indent=2)) + if args.artifacts_out: Path(args.artifacts_out).write_text(json.dumps(r["artifacts"],indent=2,default=str)) + print(json.dumps(r["metrics"],indent=2)) + return 0 + +if __name__ == "__main__": raise SystemExit(main()) diff --git a/benchmarks/ComputerSystems/CacheReplacement/verification/requirements.txt b/benchmarks/ComputerSystems/CacheReplacement/verification/requirements.txt new file mode 100644 index 00000000..3aecde93 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/verification/requirements.txt @@ -0,0 +1 @@ +pyyaml>=6.0 diff --git a/benchmarks/ComputerSystems/CacheReplacement/verification/test_validation.py b/benchmarks/ComputerSystems/CacheReplacement/verification/test_validation.py new file mode 100644 index 00000000..247a0259 --- /dev/null +++ b/benchmarks/ComputerSystems/CacheReplacement/verification/test_validation.py @@ -0,0 +1,55 @@ +"""CacheReplacement Layer 1 validation tests (fast, no compile needed).""" + +import sys, tempfile, json +from pathlib import Path + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) +import evaluator as ev + +HEADER = """#ifndef REPLACEMENT_MY_POLICY_H +#define REPLACEMENT_MY_POLICY_H +#include +#include "cache.h" +#include "modules.h" +struct my_policy : public champsim::modules::replacement { + long NUM_WAY; std::vector last_used_cycles; uint64_t cycle = 0; + // EVOLVE-BLOCK-START + // EVOLVE-BLOCK-END + my_policy(CACHE*); my_policy(CACHE*,long,long); + long find_victim(uint32_t,uint64_t,long,const champsim::cache_block*,champsim::address,champsim::address,access_type); + void replacement_cache_fill(uint32_t,long,long,champsim::address,champsim::address,champsim::address,access_type); + void update_replacement_state(uint32_t,long,long,champsim::address,champsim::address,champsim::address,access_type,uint8_t); +}; +#endif""" + +PREFIX = '#include "my_policy.h"\n#include \n#include \nmy_policy::my_policy(CACHE*c):my_policy(c,c->NUM_SET,c->NUM_WAY){}\nmy_policy::my_policy(CACHE*c,long s,long w):replacement(c),NUM_WAY(w),last_used_cycles(static_cast(s*w),0){}\n// EVOLVE-BLOCK-START\n' +SUFFIX = "\n// EVOLVE-BLOCK-END\n" +def cc(body): return PREFIX + body + SUFFIX + +def test(name, cc_text, hdr_text, expect_errors): + with tempfile.TemporaryDirectory() as td: + p = Path(td); (p/"my_policy.cc").write_text(cc_text); (p/"my_policy.h").write_text(hdr_text or HEADER) + errs = ev._static_validation(p/"my_policy.cc", p/"my_policy.h") + ok = (len(errs) > 0) == expect_errors + detail = errs[0][:80] if errs else "clean" + print(f" [{'PASS' if ok else 'FAIL'}] {name}: {detail}") + return ok + +all_pass = True +print("="*50+"\nCacheReplacement — Static Validation Tests\n"+"="*50+"\n") + +all_pass &= test("1. Normal LRU", cc("long my_policy::find_victim(...){return 0;}void my_policy::replacement_cache_fill(...){}void my_policy::update_replacement_state(...){}"), HEADER, False) +all_pass &= test("2. Forbidden header ", cc('#include \nlong my_policy::find_victim(...){}'), HEADER, True) +all_pass &= test("3. rand() usage", cc("long my_policy::find_victim(...){return rand()%4;}"), HEADER, True) +all_pass &= test("4. fopen() file I/O", cc('long my_policy::find_victim(...){FILE*f=fopen("x","r");return 0;}'), HEADER, True) +all_pass &= test("5. std::vector in EVOLVE-BLOCK", cc("std::vector v;\nlong my_policy::find_victim(...){return 0;}"), HEADER, True) +all_pass &= test("6. new operator", cc("long my_policy::find_victim(...){int*p=new int;return 0;}"), HEADER, True) +all_pass &= test("7. mt19937 randomness", cc("long my_policy::find_victim(...){std::mt19937 g;return 0;}"), HEADER, True) +all_pass &= test("8. system() call", cc('long my_policy::find_victim(...){system("ls");return 0;}'), HEADER, True) +all_pass &= test("9. Missing EVOLVE-BLOCK-END", "// EVOLVE-BLOCK-START\n", HEADER, True) +all_pass &= test("10. Budget exceeded", cc("long my_policy::find_victim(...){return 0;}"), + HEADER.replace("// EVOLVE-BLOCK-START\n // EVOLVE-BLOCK-END","// EVOLVE-BLOCK-START\nuint8_t huge[100000];\n// EVOLVE-BLOCK-END"), True) + +print(f"\n{'ALL TESTS PASSED' if all_pass else 'SOME FAILED'}") +sys.exit(0 if all_pass else 1) diff --git a/benchmarks/ComputerSystems/CacheReplacement/verification/verify_trace.xz b/benchmarks/ComputerSystems/CacheReplacement/verification/verify_trace.xz new file mode 100644 index 0000000000000000000000000000000000000000..9f5054236be6fce42ec5c1b438a6c47d7ee3384c GIT binary patch literal 74756 zcmV(nK=Qx+H+ooF000E$*0e?hz`w@?fUV*LApsTu0002#imq4x{~t|)T>wBK1HcUH zn-mOt#Vs~3a?vq1WZ{`TEO`#v zIt06oDx;I^wwF$?kpOdHW$F$1*1=ZlS2G&|V``Fj8TQ|i(!wwVC#-5QPC}qp;7h>1 z=C;N6>{+VspYrH8g+i-{v?@t|HQVMRT<{DJm4d%>PFpG{0R<46M0%0kYJ zX4i`@kC;jcF_o4hJu2-y2aL=e{18{6rCC&t{d6&qH z2t^~elLfHW#+n}6Wz%`gFPCDuE>VO$zRti?inw5oTE;2bg5YesnpYueo&_vV$oB#D z;Py~^?ude)vh%!d7#D5)!t;K8+$y7#4QvQGZ%!Mt0n`;4Osvb22OI$7VKZ|0;3hxL z9_%g_NKWjhr3D~vO!tjE(#k~t4#{MT)hz!&Yua~E8j=EM)CK(Zya(^LDNQ%IZkQ($ z7NWAYFbT(qvVpVFKhm5377t`8D?hS3GBo%Q)m#jCAs3 z{*Xy>3cWK>gnRS%5GThkjh>>-Y>-uF04L4(e5|Y<(66HI(}JWXzapyn$7@i>1jbUr zA`}2IV=(}O@#ect8o6ieP#HaR=Z;&hvG2&t<-DiINYgR#%-CcXLcWy`*EJ$Y4;EO3 zmfTPdf#fyda1T?n`F+mW8Vl5(5zXRDq+j9E%JYK`)_!p8{xrPd5Yo`Y4Tn7zbfVk* zC!6e|%8>0!E0H6Sd<^Jm7fDC(R9igYSg&JXb!GFi6hRd@2l@&Q4&;S(%n|?xz|KJy zepGFH)5MUWnG#Mdt9`qj)fNKISr6K9@cM?WqWEXd1 z*zR+}!4;?)5@3vq<^f#5>umi;rjEcF36F(QjJlc{hc&D;;%@+mACA1zDm@qlfL%r| zk&zKvHk97hUH{pkLBU-Ux(!Y1hkZ|PuPs8Mv$JHS+a-z^3Fq65GK%J;n;Dfl2G;UBaz{6eP6&eMv7EM)m0JcN6v=|#*}(12I0bgyam--1dw_Jy4biolxvR15 z?Omh!mnzcwfbsxIe&P#ou{w!?gY18N$y6w_@dDa8uT(;%pQ3% zex|d?jHNQmxr;vwn6*IwdEBiP_?trykefZg__#qI8buv6xQ><9i}Np{6g zD6!WSx)-i*gY$jk6;>Ui_dX)2t*_lA)wuKi{_|c4QUbk@fcr74UN4DP6s016P)G?S zZyk4g@-j|HbC7#>)CS&W0LCm!H4ckqPM*5b^GO(Uz|9A24F|UkT_S`}PHjeuvhR>^ zZ-(<4C~8-GEGBM*HJKr?^h4U(o!r7Dz^NA=|}xj!H6Wu-oh!R`&qIXxBVukv*TY?&f1OA3|3;)d{ojIeA6#;Wd9K5 zBx9K2F!z2QU^o;2-hKJqUD+nMxxZiR>BBH~?nLNJkq|vzfqBLL9dUC%xT#8QB5OJp z7<6M~KBIw5(Zp|l%grtTN18(Xm`GT;$W8~UtFK?FlI46Lf?iCwlzpm3 ztmxEp0u6gYcK`{oZKb*>BRlSmaC}c{)%pL*3{XJ!xjv;MZ`%jnSDmPjKGZF#ia5UTc9z!$+;O{bkOl90HEWPpOc|BN z8qNL=wS=%GQd@$$NyXPh^Ot4@wBHU&h9?U8>5m!{Jg*++HG&`w;Q8-*4{)t|f=YV% zZ|h#QpL&KEs{I&7zw#(GlaHOB{v`@yv}sW;h)m*{VrM7L6@u=4)Jl@5F;{#7IYYJ* z#(In6EuZB_545q@LiWll2^e$7R)D)vlPgY80fKQ10Hh`XdTB^*wvL?dV18TJz!!2D5^d|E^2 z)_t^v3TH$2Nu-c%@;7t)6DfCm@++(6)NYo%StY&ttpgGe5`$2UwIqWB18RRYqlTEU zRiWdgzs(f~>_6oJpZpL_FhzlcjYr?j7Ll-%f09!+3 zJ?4|(etSndQ%EKH5)C;9@`ZslSj@tE<^~&Gi{03DSSILC7muebxKMa}m!@f4cR&TNy+r1-(#FFO zxE$eopu470_6Y!|$li~~)VNcgT*AgXVc?Q19%#MVFkR|@%lER&G<0{ZroS>Ey&3zD z%%4(~>@yy*aCir6cU^RHo-7^dZfzB6RJg}_$E|&KFt3DSy<%O(#P|2CWUUho#4}Rx zbxv3Y0%ooCU)S6dFv6ZmQ1%i9>=GQ@VKVG;F)n^2n5CkQJiMkk8f2+$_Y$$ZtYlBm z_xCHAV-2S1i$Re9Whoc=BHCf<1e>HgV&k7{xVNFLbftvK{olyX6)@$CN4g$WAE}#W z9A*6w6E&h$YFns5$9}|7qD-zfXZhkv#9f&^pgV=UFROn)A5Mn+`}w!ZZw<> zjK&>)`s_`@WoQcSwDvlJ!AXjiF@8Odo(QU8{2u!uf3RdU^4k-YoRU7l;ce}UX2ZQ0 zSsW(p?^@YhC-`pUY9kXO+cM^aO(shFY|oF}iOTi!LIYlR6q;nnM>Ki)(8k6b5q}jT zyDGYo51!QhDMy7Qz@&&#zHvZRB&2p{X0DA3*|{B*5dzP%4fo7m#>k#DUBdpodxSe^ zy+PABq5)=m0`cblCTbXP4^ilX3*u(AV0OeWV!2EJn{N;$VduEIofp%!%0u8P02<{g z<;Bd6*bTU*36Eh?Ht((!P>-e!1gcY-)P-G;fzR=Hg=ctuiC1~hWt zgHua=XtNcnuy1m+y{b4+I)`uuw&2T>1;O&P_K%yEKMC3YK93~PZrKHR7I5_y(J$$Ux4s1^-P*xpR2%g9p3r-C2t*(WW~>&ji`O}*-h zg!TKxAp9b z+hv%a{^`z6!^6`fChhx!dmFbRhC|IAYc%F78TayN2L7i%_ph@Z?TObzr{MRT{F}*> z^iJ9foHBH$JlUNAsYI>Y$6?G6VuyQ;;@}q5uL+Tyl>&Vu`N{{4?;enf&yyA(iv>^< zG|~yN4=!)*cR;FoZ10+&fx|{IV%t0S<+*3$%J=P}m&RNb6`A=X6oR+S%`Sg1GGrwG zAtXW)8@{j)w759e{2>Hnq#UTl`vGi=00rEnsa?KFcr4Q&2dk8;j_Qt462gAU`{z+b zY0_<+*BIz7@&hOAxa5jLfPE_`NLhA+MpVXJ39OC$qAX=7#qzJ z@WZ?~1Tn2}`Ci%g8iDmj%hu(E)FN9>SX_vYh!e_R2NoO#5p@|KTT`o2K|6C-xvF-Z zgg@>DHiRDqGgit+MaG0AAlJWe)~N0hj+;8FE?OxC#E#fjaWy4Hrusu$CgNo}$yW zwZ(wXhnW^XMy4S&?-@>t4$4rQ6X!tfbXf$cS9zvQ*+bw1U~x5|ER~u`o8DAt?pnOG>YOcJc9{M}gW3aZDG?5uNwUgHR{cwlV!_mv|j>Q(a6Uz_& z*95^}JIsYW1ccwp#SK2=bFZX~)AN3t)>8qPj6oaLOlg!?kN?Zt@3dDNZ-45>e#_Wx zEc0k>k_wzZhsqHutr4Eq1d=9GunCO!vRjl~3^V^{`t$9E{W~18C3~U$rqXhkw3xkg zDIxniR-V}5V&%rY>od4w8y2>9@NC_Xzs114ybur3QLxy2NVZ>< zZeu`@-HtsfEw@mke@qx;JMdG9$-2jW9r!YLV$Lbv@7;cpiTT%>F_;%{+FlKl7%y1% z$6B9bCo%?=%jCza$?Amu5td(Eo|*-lVcyVek_d?QXp2}}>ez*`QdCzm69#eJQ6 zk>bylM1+&b95ddBdH6oEVGKL7*XI8LrY5q`I5SAbdA$h%{ALA=7g6Hx@Yd%mg~5Sc zmyrHY)uy>HPBxgJjC%9d2nzD(QeEUe7oB-z&0yFWqXhSA&nhM@NedTqdR87{(FS0Ks*V_5Afe{RxT{&Lqm0qI;?~v)l?VcH)Zy(Sl{VniVo}l-nOWd zEM=e9EO6RzbjpEqXt)X9ErF1X=*Swl2j^K}gaEjB1cHcM zl5Q4&>S}XOmL3*DK;%sNg)kRc)MZY}rtskc^a5$aBH&iX7PxXMPm=!=o{2iMv=62bJDkk`+Yl2ckepFkMy{EWA6=A{gr1JZ&=e$0}JI z^`);w9i7xHQHUJD?r41OqiUd9FlsVksh+K1tJ_JgUREVzbSbBH7ZDTGoZTHv+<|;B z53pWHI+5;PwnSFjtZ*peOYV~UZ{iIBRLsdHnOLM#IJR?lVjPlQf`MW?pC<4a;c6JOd03BL!t`njW2SWTISX=J*R= zNF9RF&usqohdxI`OPQi34+0i zHP-Om9AFSZLRpU+U7#CoG7m$~n=fXUD)s{qyUZSuvh1$mzBx`#4TMI8N6&64#QB~$ zxNC)Iasu}=HbDXHCms@}eYTOROdqv}>IZeV;qTKQ2*-~gR`8B|(TbSeh6%{48h(BV zdoL+XIJH@LJa6_j!Hcq;5aT-M5*Cuz1cJq^LkV_@%;fUU$YVvFNPUVrOt+=@oH`02 zir)hax)PDBDbi->|A%1T-;2m2jt@B@xIn>*f~*fGvN`6r7<&%@uTWIxY_eQmHt$Sd z^=4vFDp#RsPKJR@!h$;@9*0?W9JlABal_s-W_Co{Yw~1bTs{US3 z?&1AmM+i{4jLB-+&e&2$A%q3UJAx~7$387pD?oM-*Tl6SwD$P|;z(wna(|-w7=I?e zj>g+AxB6i~v@aZNfqmc3?xDkxUO-amw|4BIq<^Y4+N>xGE;v7#d`=D^ay^JmzgH=# zNu^<-k)pKa%-;oDKU{!M7v%pM*&HGfC{-veBDcLP0`@FS6NJ8e3YC<;+7cBSIWY9M zvs6`g5z865z<`ffI&K>;m4#;*6d6y8UnA($P+VY;UoP*L1X4S}`*&|*z_U2<*Y?$|1#O2p zgCcUJQ;4(s+%b*gk`w@rLXeq@RLFkAIkIoG3-#O2Au}=O zxkQuv03XRmu2=5tVyQF3Sj?9@K?2PU2T=%;;oVPhJA0Gd7br&KiyS=I(*d41)n&2>HKW zvZHZ;o6fH(O0yEc)^QIx*Z)B56&&*_cGoSgtWG#8)_)ne8-xW*o@nP~O3*N7Syb11 zk>I_Ir&=-LPd=)Du{8{OygPoRgB@mXnE+~!ff0Yzhv|f|(yTs__u#gycnin)=$vXw zLDtU`qj1by1adT)Uj@ke?0Bk$J9be8e`7PnM5<|&mZvQ6#vea2iS=9_ZJv&WYyyoG zVaC%-)SHy{UAZETmDPde{u54np$i9Cdd0C=2WW-O!{GjvT+$UnTt$8&8{Z`5pYX&P z*qd?QoqvyhaGw*70ecL;i9D7zk%0MLFizmhm5DT%vB5;ST!5=($AXG?zTje?`MJ=w z36_TDZ$Wd4M5Gb(2jV!n^~YqoYe5wJ&_~i&8_hN=YtRT^%5&~y2OvKwqzBq-g-B$S%r?$G%8Yod$-fWCt@~@*mt#+w<3RBYAre`6DY~rH56PW zdszs~vYsu)FeoXM-L8SJUV;X!Xr|+Pw|LPGH(2bH5)`+MjPTR{?@CEt`t$g2A_EQG zRR6EYmn$YFkV5kyfFGu}_7ZLqvxC#gOcwcXj3@nbJwUZa7xux|QE_8iZSg(5*+>>e z@*MsUM&}`1qXK8uT#;kmuDl`seDh>s*a_qF9H%Fgqu z&PVdB%<2nF6AQ|LHrnY%SxR0mL9&)t4kPzOf-tx8ebw$*i)}}A3*aUDSio~n1(h8> zNA<($paZmH8f1koeMAKH1JMu(kwVhDq<#ByxQQ0&_jCx27}(D&Gv*H=7 zxOpA~_4m>C0pf+P(`)|4_8+&5K-``0gk_@-%tX|(C5}PPl*&XT$1~U_Kd9`N(N~b! z<9(YN-y3a+heaUfEoY@3lxWt5egk_%fzr~TWR84gK{nQ+Im8$@&lf6o7II{PPqnc|%5l@N zrh5u=!y=UUq^Ktg2BES*DGDsZ1G3m$Uc6_bq9hHt9K9c2rHqu%-K=44G7M}4y*H@Z z*^5y52{FrR>whiSB8-PLgWBC*YrP;3V&M~?kqqEBFU%mE6M4zLR}#;cjBBvx6Oi?d7Jp`vfRQv1wZ4dZYg}H`%LjO!<=c^i=&KCDU`<_b zBACcD>OWoOj}-qpkx0?%26imTN9akWoOs7Q>V|mhVD?D|c$DNj$>><7a)-t>Fv@`% z#mVz+=>>SfZ<3ItD@JE+F6d5Q{8zXq`H2ePbr1{!;SWuGD(&b|R8N`WjPi~HN*18u zzEf~^oDsHx912^&3 znRvP_GpVC& z@f}~W-{n=p3TS+xGzdH@1U$0ymx7U8+kS0INk17yqO@{18S?i;;+sl81?o*ZhiB)f zZ>J#dEsIr_e|3P*zhvZK@O@c#Tq1_gHmY*M$I`%RiT?A)tymBos7=eonW|CaOXaE2 zqM!kv?du+K{ED#!w}Ij^rVAFnKx6EbeGWqE?!@KpsNkk?f0CA1amAT!)!Vy{J9V`j z0a5vslr<8{LXe)1(~Mj0z3z~>RetKHQEsm2Nk>vLGi9NN7^ ztbGChbGQ`+>69ODQ1lqz2T_`ds4VqbZW+^O6)`)A;$2M`a!x)h4Xo9Y=`PzzkfT6oj|Ji&Z~cWIc41l)2{8gE%lIa1H*z6KVd~R z%viJiwof|WSvD2BG%~Y(LyA3M@#xlddoJEBEG;)}8hr8IznenHATsZ#Jj!#IkwNTI zFPrSvo0<%EZs3>^hG=Hqg4!|n;Ig3d*xvZfNo19pFs|u`&wBF7l2aonI?z7X6WZLx zYKjx5)3lB9u>$^AT4KG*rS8M4u$97n%8C88rOlmorT<~+bb!UD8BSB#9C`z5(LMXP zLA|*WXMD_7-2Ucc-Q&Ih15F8pnf}Qja7K8!1k3-kn1o8gSdS~+$vWJ*^VS-~ z^@yRC-Elqy5&!kHxZ-bWB)+VX;_y#XAN#{d?&7xxb2caqX>!?eBS06=Wli=iD%>l| zVK&vzzu-e}nO2>en+e3q%criz%Q;I`tei;97uaVPc}OJ$mkvu5_A*#8Gkt!csqWTf zD$(};^|*mBBRKHSlFIDM^R`4gi?Q&@K=0`~dS;BfA-soEkXY)yg;pr+*bnOp$jL>y z#%rarId=%6=Vlh6x=^|-rJ|j~C}?C?n1DC3Bfj0)`(@=v*M!^N!AuP=eyeBPmcAIT z?c#VbOCDLV9{6XQ)2MaO<$hf|JW)#`$`*u7_H1lR%%xZa3C67Or&V`lIEySsc(R95 zLr~z(WoB3atdaf~$Wn>r6)fQ_T$01HIgq|1=?;K*10ytbg9ZgmXztR<`6 zC+nlGqVF$$=EFWP9j2nr>3kem73e%aoQ4^FTv`d5J+>~0-vL&G^9bMNpf+O&BKc78 zFEWP8RBl$C*Q z_p0_z&6MGl?VQJ(zTD5Ym>$DcYGm!>9UiNc2`6q|ldtRp8lb_a(%m3ZaEFt?mbPGY zOp=-UJp6rJ?w*m7jmz!gE09gtLJ;A&Iuaqk=213AwRJ8FTrs!q3$fKnt@Tu=z>(8?Ze>puHYNdr~}djWLM zUoS4y(LNZsJxsR2d?AQ9L82={*DTh3DmXZ>3FBNdP3D;||FRS4#HVl-^4z_t!(>_ZMdeiKql6x9DW-=Q+8E zz2sU&{d22eUT&7EQZ4zZLQ-Vp-POgoSXy}vOUEaQ;6f47?63Kb5?JA;dwSkU+ z-zBTHhSmuG#}8MSqyb-oOgl*lLA99=b=5qsv=}ha$!oYJu8(B?r1XdyiJ}ioncMck zVu`qr3B7O9vX6!hc1cx*X~)k|)P$smGV+-KBL%hP|1u{ypUpFG!FsMI?_0SlnmvsZ zM>piDV``NkZSrdnP;?{&o12mPK*5SMcmzhfR7?>g07zf-q0dtK`K$z_9CX5E8o|X* zJPKwVEzC}Rmcq?LDCCPY4ZZ9^uTXv*#%*B!HP|h^0|MBp(tek<433;u=p$F1p5&5# z35#soj)_;_Y*SqTE*E+QWE~KFg8PHthiT8o5UihOb0uWW^v)_|wd)=1%YXI4O;v#1 zC-!%(2%ApgUG!Squ5@Ri!C=JV^62iKlz|g6*q0v%BG*oPH4mUKASOp#pU0#tFMHNn z=!jV!Ur()kKjC?RoEB6}-3;|D0RhPuMjE7*k5;*4S)x=-?>#R+@Xy6Qyu(% zJfdL1FMcUiox|QMyABPk!46(XlXTfjtf76o)vwrQMce^1_w~ZZ_zR7q1e(7*LsR&@ zIh=sVwotOqMpuYSGsZMI)!{N&jff!;hT@A)v|IIEFQ4pDDPKB$J;xjCZ{R|oR&4Zh zrNHApM5q9tYcahz+}cLJyXNYQ;4%q%VIsGU7|WjV&Vb1B0eg4vQaT(!_e9xcvIWFf z4~c@?QvoX5`RnjCEe`2ze3m=u5C}f0h}41jKQxI+?G>Gj_C^=ngQN?`85UgUcJT5F ziwPYiCt1mJ#qQjtNhnaFLQP-&vRnJQjo9-!w3`q?@trT=ol4jTUZA!i`J2?GXLhKZ zJyJ6c&`=4T&av5Uh?ozhBH&YhqK}G+X|0V@*^p5&oYOERmzxM_3e%k}K8kosc90A; zc=PH8v)jTressu^iWER#iD~t$siE6lpJD8@k6kA?M&2K`5Cnhpw?z{gCY1T-4uVIedE8KT zjgqfm<;l!MLnjR6=0uh+?+?r=EPHWG+9*Ko*3^U`^(|WGf;{+xSabDm|CjE3WH`mhorPQQl27)Fq4rD!)+4z_dsRe<7iD_~2_r8%75g1-y z^5<{Z-n9>IiVSNqkG2?@-)I~#%K{t#R@Bj(UZBf$N++Q2l_6Hj_;zoghR8(AovDd~ z0hDT12i=idSP|)A^)>1)Habb5eoa7F`iS8(S_*M4Op82isz#yyYPppk2Nc<`yfY!( zRJ@G#(Wiqe7HXn%0(8Xh3RD4SPkRKXa|M6to0T}ap8+fhy`kGE^qk!Icr+IPeOQoa z+-!$`sYbT_XP~8(cZ8+Bz^J|qsj}o^cGoqkcd@R?2*ALFFs4{)+V5qri40yxEKvi! zkd;c2YlCKALc~TUC;4>y%|=^16A3npPK$ zTZ4FwZ$Z2x@PR-V=^>;obr8`>H9^;BrCE!dM|=}GMa-Q_6g;H6AU=mxsX#SMpg-4P zm(*H;Ap0G&Y0EGrMQwEOSg}vHK~mF!7zf(-hh@C+w2M{%Ymq>SN80Dzx_Z$^_Pz2| zrEj4L(&n$w%M+*eOybucn64<*pvI;eGfO6L`wXQjJfan02eo>4uoNaIv)B+n@M=S4 zt}?&w(r;Ob4$=bT))GOeqpMXnXe!p2Kf1*yanhKw2tAS$qj?ZUlniPlT{-~9J%Z04 zPXvx`E@h7#Ea}h~50lbv_kg!8nAK_@&_Q7zs}a3V#~9n(KUFvhf^@ zxp%ndxv~4uQ;oA!E)X&xJOtkF)e-8b{kDAAjsT0|M$t%oTu6qpDkSQ&)y()(t?$q z-HvtFW~Q=%VU#jVojOiWaM2~o9Oq7im-H({sbOB(!g=7VoRz5(OZk-vn4z~Bq@sp6wNjkC0mX|EGgLDM5=_BfRgqTm8(gP1<@83 zBAF54>n@Uac0MhF=t+$rkyGHsEO{_GQt}Tst*Unu5)+~}AD$0{|GCdC^;>9JRz<~a zr!L{c|FK}5nL1ZUDrj#^tuAIl64%Q`w}cYVU%S_6i)t{iM&e6x+|S$+z;JdYYhtaK z<$IR!`OU(0INq*9S^RCz6A>PTmn>PGTZ#+KcD?m~d$52D%+Fb4DS9GY>QSkZuZ8%+ zSFPt=uj@$L@#Nda54gxv*+fJieAmaqe7NnmQkttMim2j-h|c||%?1mZ$Fvx!5VPck zL##+u-k5UVs7jxkF*`^Z<-^CbPW=^X_AreBvR+-TQ5nsd2}E0<0Y1S}q`Yc^#TglW zn9UAyxH$-<3{lToExN=acE59ZL_6;Hl}5elfeF#eEyoTM_vle(4#YWkO5cHH&4zx* zJyb6AKBR-O$T%|TJk~wrYbVV!uGr?fQSpA24*cY%MvR!IZGi2cu1r&)mcYgaDeL~J z!;cbx6dNQDm_O;?PX(wV#w0GMGD&Ru<;8Naxj*di8NV*o)sIp>h%Bq}F@q~G_3Gm$w~0juEu}qzX!Bzn+abAF|J=J@ zH1211wjTlsWxsz;^svZKmX$wt-yQ)4`QuddxKy??TEmU=v$hONFswHNLd}FC0S2pG%6gtcoGO!m( z0{0iH8gKa?c88YGV`QsUlVltZkw4OepxEiwP+)f;DfBC8*7toQQ}K_j4ii?9%p?eO z)T8S*dU^UE8BvYIpZ>8Z_|&Knlkn_a{$(R`QV(pnK~_&&QjUN{T zpKb#=fV(A)(TRO?!ehS`c-AUT{~$6e9jCKgVEPEw6Gl19zQ1g=tb+J)0luP|b7|H} zhEfI{Yj9jdJPwGBTyKLS(^Yulzi|XlN80=#4Z3EeNE<9!|CY>i1_J@ z*Nnx;Gu~KJ4{?G^O)m9Z;i_oD`gJ?;_1ayMgSVsEp@lNYZ4{vE5`tk|JY1WGbf{aB zn%r@GRGl$<3ujMA4wa5`| z$3sM49&)8)6yUnH844mMVl9TymFLpCgon@pwJs$w`VgX0x5nqD?xPYKeer}pe@7++ z1nx7^b2?Ijt9&^XUHoF(f(+P)3$rdzZkMC@8fB7v9qLvka_hn8^PVUhO^9Unx>W^9 z8KooKy&{YA8W)=@!&e0QnL4QK)xG=lpP@`bd{H8>-w*jiCh&|m*5TQOSHpr<3McXO z??&+_ESlgvBv^ecR;U|3dA74(Nz%HItZICIN4FUUJSjXxP)K09J<OZMYNki8bE zJGLr3!7n2AvbI2?<@?G@p-E8_Y+P|7OCFcBF(=Dmi&@yuF}(CD7Oj`Rdd2W!HS}-D zk?e5zoFlsDfxxC~+R3>;8|HQ3?I)?bc9)ejI3pK|x4+)}#TpS7kTmMFP_N_<4aAm! zyO=!;S;zH&dJJ5}^Q8-Z0R~zxkZ8*6aZF2IsGRyg7ZJg0MM=R!E9~5;IP=lFSmub4;^gD;PRG&LFoDMgAp)2^ zfJKrTi{Ihh;mpk?b@9zV&o^z2zw=;T&r1)knTo7hkYEqVFGJm$D}SI=c(NpXU&O~e zJq*>tb^n@d)c>VysqQ#YEWVfJ-8sS|GFp*9=_qF(8A&W(0yc6x2U1Ydk${lG7s70@ ze$~oVtx-$306x|Ef4{a2vhmBY9WSU56H($p6(~(g&4Lpv_{K8x&-f&ins2(FaB~hL z02!e{R2g*03zKu4hu2&@iSW8P)b7?ZbY0cnitBZH3Aczx;@O$c4kTVp8|)TC$F zu#BA<-&%)#q1MGnSOKq6!W`??wo+w+KVeGq9_@E#&egbB^wThx7;O*%`w{LD+Bl1n zu717yEQ49}$MF6Kdu zhB|w(r3E(fT{(Ij+0BW%qg;F>7I_+T!_67*lcz9>?Kix$ee5mj3gG=tC{O4_050VN z{xJ+D0C=}dwwR@{5hPk#!x=$80=Vx*4p!_Gq#C-|Y|^VO+SRQC=oP~=>!&ON6EjINivkHa|@{vJfZH=k5_Ibw`8vp{wJCb zf2ls$+U@AnmYU7Pg=cM37F-Q{o|eNUF+!@+h~sYQ`ekyt3~JR2(&B05Y`98&iq^FC z&%!$L&ok4(=-4-oBP2t2(|T}Zcz~Z79xKNJ+S(Y*LWl3P7v*pkcj7ns9rdSAFg_AB zh)d57E`%u3*oYx4ok9|7Tv{@{b@fB+!?krl`(hbo>6yt2AG#aFejRO$<6HNd>cYUw zwQOm*_QS}A+s16V5=j3Kh*dwiJ&t~qHghA*a@Ub{USYsnBJ(}L93M*`sTj`g5mUtY zx>(|t=7LAs*OVd3q?gsCu$ozmj*`c73JgExbtOl&_V6B}$)T$X==)qJb4cwaJ77@x zlJ261?eQFkCXRkX`Gc9&k4y-5PYi$QpfTp%H!o&@bAuKITIL@JM&pNE1(gYqCu6Zh zY?zqVqoQJu!Jt*D2>$Q8>yih6){PayPCeC9^Ft_SZ_|Ml+ydxN_K+ubL+FCgYQ5{Y3< z)b1jH&Y%_J{n_5QhjB5k&P1zIR{gDq!TqvqQkW=wDr1DBzO3-o6@`1yb} znE-U|u^~QHIP0Q>DDOLN?4}W(Yo51P9U#Z(^pqM+x1TIzNhBIUv7q?3 zWNS*)i{?C`CF4uKhrHoTTjg#R>kM-|RvKnz9jc_a!HPQSysPe$yk@%1%Q+MFot<{H zjNqL|(xo6Xao~(E(GtKiDzLEp?NPd=wZq2?jdx`g_3Y1D9tT_f!4tt#Ooo@XK)ppW z83dlpK>oUx4X+(WDv3z1Ms!&__Lf0BgZzi^Qd=uj=A;09g3+{b*Q{o)Ov@&oAN$<3BgK7N{lkU6l(&dK znMS7@qbQ{w9N+k&T{f7_s86ZgT#6Zuv%0Q8XN;W#utt%LdUGf##~sWOWcpZa?iFK} zfqq9@@$!JhR6!y5@Lz+DmclK7NK1mLuKc%pNl1Al{;P?aLB=x};V8Nr0>Hgoda!1B-*B)jsmTs z($7E?I}hu(N90P~Z}1!x+Q>&^nw#`hBOCHttJrT~49-&V2y>7Yc>4ad7-ZK|fo>CK z#^ZZI$%e@3zqv@nQ7Uz^&5~$LJfo?qOJlp_pdF^#3pbio-FGmRCr#h-dK5{q67d0L z@y5H4l(7CefokyLG+?k}neXFjF~|$xrx@{xL`YhGdLZ60&wl^@iDY1-;= zIVKSFNY8u`7|4tfX>UPT{2{Be0BnbtcG{MY;MdN7UT%Q6-i(+80_|8&lN~oWr z_GsB?iVt`ypRoB>p=cbfPZ)lC>?2^z^y6apxAERHJgN#P0YUet+yg}s@>g>2Pm0O7 zK{5>uH9m?Suuz|3HX<37356m;>#UhlBL2EqRRoSW;62bWBq~(e$fcewNCLYD3KkMu z!v=wPe6liY6dQ*6uJb31v&4dz4aJx} zzyn{t-uCridwz5t&Bv8rul};9wpD*+x|D>Bw4E6k@F2TqcoEYyyCJHCbG;lNF362$ zG@Llew2nXQJY4ezHSthCu%j9KlfuMor`4xK6Dh9IWnN;dQIk~ouz)rwI|aTOH_Gcg zv+Za$f;mgZzBbE^mCvfC$`iT|cuu)~#PkL5e=e$mYH($xbYDX2cI6;77FLLCrL6b> zJ+8SRE{ja6fi;3}`Dh-rmc60`p_d4gTkDjmV#0%b5pXuq#TCK&RyST;InNY9qe8h& ze~>%izKrIZ_Il7utQCGX!R}n7t6;{B-rECv*^`*s8`JiS8hd-GFSd2AUh=!K$teeS`i8#{=MOadM=FJv3-QcPzWuUZ!7DIq^IB0)ID6@T z+-Rbx>uP-_A%5Aq#`|3b1nNUNlH?Zdh$p)3{hZ9}|RN5?C3n5^PZ zVd_S<^v%ekzW$OyZ68~y(MB=X+rdTi!QQ~l?U%or|9w{ zALAuX+P8&|awFMeh}=iirk8}x)$EYc76ZbtJZ$BC`xVRbUGcc}{XSMmU_FS-+y}jC z5U+Zb;|HrY3+cA1Nh%B-w3Ag+fUJ+cfOm_tBW-p}2Gi(@sGy~-_#z-5cnDoqJQfqB zJ_6<^@V~dNe_ymagnoUa7M3T^Nb5 zf^D;*Wa_g$6PbQ-HhDOnyB$9bV#3hB-CYy&E?vm;aZWhNajKVjxJfX{b|%Z^xGIg= zIo7ztQ+Qp0{<2)&XndUl#xci|?$2jiFd=hxc4<*w zE@z4~SePrcE462+E|ca%KkKI9Cb+r|{+&34ynY`1T*d=eOoMgR~5yZ589Ia)}@ zp_N@J2G<5k2do(e2BnNGca0u(8mePiRMcg0SE*|shxS^CT4_g&2a9kDBCdCpl}-WX z?UWaM^H&xUn)09=SB?cJwqHmEvqD+)jq}=ZN_h4AKQJloRYL@(4n|bm61Ov*!%X{33gGh_%ofYB(0`*zCVT$ zfaOngv2Au9V9*RlREat{ZdAEjTpn}ugz6hXVCC%7yY;$OW#8yx8U;@-dsH5xRBEJ$ zjsy1HwHQMviJK^Zz-k)+z!$JwCtVU)0IZ(D+Lwe$UQQ09*p+7bU9z{9fT+-QN#HqI`4@%G$`RPkoa`DlCZD~{Fr;&V;3CIyOp42i1tBemd}zq-$*8Nud9j%)F1&-DgYg!TY@8H zV(=3;q0%7sppIqTdJX0}A#wV0=zGPOhs87Se-580o2s1(HNjI_L*x*B3tl z2^)6y@(l|;NM`r0@Ky*q<7K1)E7-3Z>7BuX7oJ8*htn@$B7!OxM`=kjPKS@z0w$sZ zX%p&JyVKa~YyK;p$Ec0pVFb2+WKh9S`}>>gk366MS|I5U#j<~& z#L_70#@VOY>_@oU9RO7a>pk}cmf4I5nd+BxoJVwPk_UB- zeV-6t{I-seKv;p_V&|%G7X^HQHIJ|2+O|jJ z(r;Dbe3Yn@yl|6N5yMm@%tAfSpe^C*|) z?2a==lJJe$A6!;tga9>koH6x0Nfp}bgkvx*;Q1K-(jyMY;=CETE~~U5WvF3)_N?=_EJGF zav6!mUn4y&`T)!C}$D6jZqjN3JoTd0< zKcQPbb8($gFzeSM1H?OqdEm{8_6l3jGS->}RshvpBDFF>7VTiiZ(48bYC^P=!zSt) z4>qSRghtado{|li#JDzLwdsL9e+2R4B+QHd?D%rnCn?Wj{}My$-lVwxgI+ z_C)<^UP6q!8Gkyid-?E!h1H1s)r*7X3oLQ~EkztzjG>@QS2WQ=Z_A?NFNH4`#|=Qi zK#RImeT+oZD=j8SLZN%8LIt};zzuiVn2wOju0~DfG>qZnFgF5maDw<8Y{jC0V| zB`JKlS#9DF0SCxy+FoJG=WVO8Q(>^mrFXX(7l3pZ`nq0(V-ZC%}>yMgPjR3;G#p^m05_IqkuioLk7%uc97$13)vEOT>{O9z(Q7a zh_n1|I!J1Kq0xjUlStVK&@&0P;GD-C879&oL=XEQH;&~3`psQx>x$2dyh^C_I}cRX z%3FIfp9FF=u$p^3QWSY-k7ElAuQ6&PeNwLvi0o4W%9*k%g8)UO9g?fHCAbG|9;&`u zg*v_9wJ1t6a6e05pj$sTc>`pD?g0U#%btwmd|?-HDn0(_{d?G6s7eGfY6D}OWJd$NdZZn^Ow;*L zyeqVx0!~N6l0v(csPDlmvXIGho89jb*?4@Qg5rOm`*XwYA*G+;N-H zOi39kYj(@c)DNpl&#Pl4Z%)hzEq6?Ht#oAG#ecLOy8SyW0$9iVx98x|`}UZ70H67R zvqhw#RC=x(E9AaFIGew5bMOy!lj2QS*~yQTsGY?UN#jq)V6onhnqY4PAVal;*eo?b zq~!Vu1JQw``RZ55UPAfbo6mzvFdaR$5)QtMq}E5}p3qspj0l7h?o zCT?w>A5-aO768oWt53rk!;xgo?b$U&Qz0HXUrS3kE^zdl|+JSxXY%j+CqRb<@WCRBG&C%ku8YVcjUKGVr zE5}!zQ!U`s`E|h8E)zP_3d1eD*)dE1<{nKz9ZIHY#MOkZ4NgC+wo08))eR}VW0@=l z(R?tvKi|Yu43z|cq8H27+vGQpNCzNSt(ZM9!k8mA$7!{R5&c6Y>4%?y!_Kj6#|4;BtQM7tfFG4XZQIcEHq44j}<8ir1;o=E|l zj@L=q*%dRgC*|OBz`-&RLsH{mtev4Q)}9O>K2?gzxQ!->9V|-AY+_sI_r&;DP3IKV zocEem%ZS>lDOS>j7rNuhc@(HWUIgpmYf`pkv1h zR%{?TLVggF7WbF!H)7q^6nvc!Et(|s%zW3`5PQVrupKmz+M_pP&c267izspsm8nMZ zmHmDsTnpRZGU>5e>>?Ntam2p}O1k;4)r56dHC99Q=BazjsjiBoWD|y%#!AFIV#x2+ zOS%KhnT=xphh}ZxYKzr+M`1IAd8L-EQ27KdI*9o6PU>*7xla+{y(RWVAj0F}Z zKwMOLTY>SCpv9dOx1F&*v8U!2`4u8tOR`$(jDs{qQrn3P=n5?TkcrotA3ZN%<1ed1 zJzk^YWGwjQN@1)`KF^2+OSTvyfjk@VJ$T=i4|d)ffl{_H)UvlIjSnkDs@JKzJnrSB zmg13$kF?TV;1Srs~PC-Kk3GWA~SpkT2i0Q?YdxkW4Y!T-ZS8Se4tv?zThMm zfBpk}oT$5fV-|DF<^>uKE@%Q}zz1O;^qu&G)BAf0hGUYEJdd^9YQ)Q7vuxN}ZTaRygTL4XBp*W%%?%(2I`Mj{{2y8ulhi-4 zDySU8K$21Hx7Jo6QkZb}2>k*Rz(L`2y>9_y&OUR{4Vb*-5wc9pp$SAUB%g#=^8!9= z{Py(XB>5bjugw0wvnr~3DC9M4yV{;wWAyjG<=Ncm=?@n=dIYp8vubUb!wc;DR#(!J ztLP<3A0)XVR6zxwE-5OiKEW*dM;G4noi4F;k1TQVRr5bwLO`WK-Ac;KO&>mt;o*ih z&r4{pJ-h3klRw@3$=@p8bgFa^QN+>MVDJM9W-DOxeTe+~$vmFK%6+ke<1{Q|KDnxjNWqX((Laj&^^;_?i0~e2P{n1 zS8)o2<*;ZI#v~+9Lh*;NE+LM{@&}FhDTy9LD4Y1Rs>V)j*8_;2_8>Sn74Gc!?CYK+PZ>x~4j7JkVk?*cxXP(wc>_ymMZczb8MHd8X&Go< zY$NGG9KAcG%nS8TXFPXP^uX@<9kFFtRNp12qp0#!l z{AqCGaO|51+tD{+y27HOw{oB;R028fKGqAEJS@wTWMKKY$J;~*;8yyNGyIZsydg|*G1=<~ zWpuw4AbxuzMTE9y0E~S%i|q$GKJUQ9wX9C*)t{Y2L|Ok)`2 zHTfnfVY#QaAHDcUZN;N2@Kz+ENl$qnPR7m2k9>b}Rs7KmPZX^~>IfZZGCjnP2wPce zx`j`%RMdJRO6$uY-r<1dU9mivAPmUI=y^kaC9$9COsFpRlwVRj>e^Wl?pYF8$j$Wf z!_Rk@Q4kJZo1}}*^f=zVQ;dkUM^|*@)cM!S5*3%P$kzF?t3Defe~cyjdRAEI?%#|! zQEKw9O1fwv7NvNQzOF9-(aoWpsaqa%%yzOE#>LXk%3~MdEF~%fjd_LS412h*z{EWb z?LEW4&cP9FwLw_HHHzwjA2l0p)Qf2wpF-VotgGqm zZQi$;*hbyCY}3;4CHJs&66vkA3X^wg`ds0|Wq2-q0|e&N-B8}~^QTQ7C9T=)sVLQd@?>bD z68A9OSvMECDT%=em;DG~cf#%B`P8sqNdVMy~v-A6C|0^!^iC2h67w0}Dt}Qt* z$5C0y0Bx~7?v+)-YI{XpXZ=EOJ&K_?Po+XjA8bzGR&58(Zfn2qWnU+B+u?c+nhLd@ zOc^C3-L(l*-~49vrL-!kqiB9pHh3IwUS`e>&4DbFBzEes=NVu!i#Ijb@ulv$3l~9_ zgQx}a5Fs)jQ2AD|yH8sCdTBD}Y+`BNh8eey<+5ip)12JTI+P+Emr4vImGQtB93F8C zt5&vP$~HZ?!lOWGIX7;b26{8*Z2ioem*_|VisdxuGK%6%$OdR;;H#yiv-cAi`rImd z9B~J;Sp(+dRI?W!xy6An1@b;bY*Qgx8vrF;=@3`nAl2HJ`wY+A=uXFr1z&-k@IG)aFL-&yXv4xz9!+GMW zrn6`h2#vBF0YJkqXzXs)*;{XsT&{~XPCos%`s6)(Oc(e+_>CokpoE>NpIs)}tDzms z$uaO#pN0u_*f;!c=q0_|;B7eO-Nuu4>vkMm>dwZS*4kCKpXms8%A(d~v>&jf%X6_CT#B^W`2t?G?B_p)BE z7aJQc&UwOrMey(XA-yoLbL;quTIe%$COKeqvBnD> zz#$c`e$R$!lO7tMb!U8^)hl11=MyCq(~S@${0; zF=uC4^t7s`PvEH+!BAUD{~3q4k0N?3E|l5+!&*4&s9YFxatz8C3-VKtO&se{Ds~~Q zhUCZFHL7^LM*Q&R*Er9s_2)X!0p0kGMgjiJi%_y7Jtq`>*Xt)q4OD@dQ~PzGedo15 zB7Sg&8PiqUdJ>UpbaGEnqg&-xD-E6ogSeg@NA;vIna0Er*4`-OHp0ZAc= zMn&2Qg$Q$qn;ogmykN1H+wQfJU_74LDhE{?KGCYjC!)FZFL1LE19ygWw0jmTP1H&E zu}2E&>`+pFC?l)K7B0mOrGxosxKEzde5a+|5ppShZNsf9vmepHk2T*9AC_wXC1~`Y zOaeb(SHb?sk|jsHBMm5H3m&J93FtfA5H`vW6XF4b7mEFM`wV>}zb`NR5lUBIiY|2h z!tq|n&jbQ1ph>+E3vqA@c5{|_X{(l3ODKUxAyl=-lf#J%*PZAm_B@*)$}KLbp}P#) z{~QbSbhzQsGHHGE_y|9x*M7|27KyHy=Kj2R8p}6N}PM1#xJ(&sxIM~%-QLb3J9*>o>?~5zY z5J3=myVlNyMgv7$SPh+V`RE{wC1veD9Cx?l!TVV2&wDLpSfh#vW!z#4Ds-loT#?hs zsca5A{xEc=prf0k`@zrAm*#Jh0x9DK1?YPMcMsSuk!TV_hSbzi7^3mq+yD9Sd9y~x z#qeeQJ)^2`&Ct;&S;wn^PxUQAazE$@Rj5(&|p ziI4UKG}S|`+-FCAfkgO8gyGf+y&)JpTPHhL*G7cy$AS^S+B^gK5|g6K0vm^-gMgZE3gu-YOpyh~iQfQ;f&-21LlE`?peU$X4uQ@QEg1SBO|QS1a7Nc%wtGGw&JIugQ( z>wr@P|8)Nb1X*}HWIcLkNWb~m2x z!Vz1^J7RMxWBpua1!N?tr^h4by4jFFNvM8Md&-lzUTjw8pv=truL;y-v%43YwYT>_ znu>Np*>S#fzMutM_gKxjEB!i_E?BI58L?{@|C7v!Q**_6=Q9T~#>w*M)^w0-S3t(@ zUZ)XXt7y-ON0}3bdhFfdT=Dn)KMIm_>-0fBbsNjmB+GVGL({{F+^4*1db?M=do;3! zOioIa&R|!}%{CexAA$+X(vbxEr5jJTQ6w!RC#lWBSAJ}``SU9Xla!K8{nM@P_WU|& z?EELiUvz6VOD_?#;#z_0sK;me%XtLaPkv-CwTg`4#~O4W zrF=Dx-H$}pjHx-|>SAOa4cI`q6+s#6E-$HirS8bHZ8LX){A}9-c2)^Rh+txCgkJBL z(E%w=m?(duD3pmAJ#ojALQ1NDhXPd?0u^;X_}Q*yuG8^)JD8Hz!s^)pl(FgxCxA7kd4Up^W~5>g4PanO zaae!YrcU>^ZA>l>=<{6b=eNe1daC1?G_nRCj4dG`%s)OvzKn;U6qEueUkT$$Qj*p0 zt@V97a2u;vGZxt>fTbe%YdN2J-IrJKgbyv8wgpruGD7fZjbLqF8FEBQU*}N+VKso` zBx?U|ft8C3&!(Bv^$^#_RB35?t;H*;*;7&{N5%5yP~Djz5((og4jl`@nV_&9IUV4f zHWf|n7DELWkYm`VC0?C^w8myHpgmxosDHxedX^g18{64^8P;(J)dka!_xT5luY#h? zJeYWB`~pWR{eey!%qMUxO59<&!x`xqSg(}-n}H_6;6k#bk827x>7R8dk7)Se)sulf zmckFc;WZ8;V+pot;5@zmhxJCPeNdvcx$PY9& zbpLTVRzv3SDu*v^LZ=(?It{`{pRxWGbkfI7=HQ%uLP?&>NNIxjM(ce!Rd-kVCI9#} z37ZP?eD;IHz84fXdoiG)@egaVQjoaksiw$dibZAeW^Tr_g>6;Ks_^K&?TkMZ*?BMV zefCkY%8GwxP%Z322#*0(Ufh^aP290-ME zM%3SUJZkS=wLNbO+e~4nIBu_9k$LlmW~|jfF7-c~=!b93A{t;?%0|`(N4o3hRNn^MxQA z>CK&Ilw*3{=2d}Iev3n({5Qj>%U7VWs6UMAZJqh28uilGZdfG=tmGS4n|R~<&i&+-4tgp|w*N2o%%!b+k-z~2QucFS1T1qlqD zRJ}(B?GFvWxWC4J(-av-EY3hTR>aNDl0vzK%IJ+c0sIIg)4VGNI(XG`qb`ixtlWF4 z)-?E(vr96-)!M^e$#aAXyHdvhBz{^9DBw=tmjS`GvezvxtJ2HO%pCcZqo~a~#_P@mfGq{}Mk;H)eC1Y|qkL^>7CQq`nKD#FlAS(M8c#=2_#Gx}?s{oXu5kghw)+xJAFTr@ z*Pd+@vZ`jFR4v9sKy0jgV#TMJ);nr&21s1=58As}cBFuKah3BVxzPka-5_v$VSg85e+|Gw?$V3&3@HC*K>~5W;k2*}S z(BGSX%HquNlULo*kk6gmPM=h7gv*;eUMAi@Yi>7_xK_$A)EFO-j6+!q=HF zW&y7~n%Qxg7BGcIK~tuZxAmXH4sy!VtZ{60i=a+%=YQf;YydHJwo(%E7+5!oj19IM zSA<%2ZyXBo(&HvpbM@z5nnjMd@Z};U_@7wHshfg$bS8c_NFtQyo@Cp8_2xbBikGFD zlcKt>oekli!@PcP==2%zC}x-%Ouw>CIA^1FGX4s7jxSDBaSY}_(@h-ya~mltO-`HH z?^hjlo+;%i6I^qs8!TI+8Xl1s`Aj4qr>`)3YyNv@zaTT?k^g$kUS?J*M+DJgpg`^i8BY#f%o)5hn2|~X@ z-Z^k10i;o>Ms)vvbjKs~r|PcIu@}fwPR~T!SVI^QF<`4zh(<;3XsJQig**aT6uGz} z4w~(S-K;eF)X%fR&%6M5P1WbAfJSCG<*UnN!s&qDG#np^1!<7XN7p)xC=P+3#1aOx z^a?qpYobmV>?ku(5K$0CA#xvu&gKQ#7xCls#XISof9Xh&T_>Ayr(s-RJwx?4g#(@i zvP(=c{z_+rm->U{;R46`i`!;q1s6ED+_StSvW`D5dER{wYFMdlDy0tY6_Sgqrry)3 zyVm$Stq~H-;y2LZi#=$>qBQNZ3DiaEALb0cMgXbnsXq~$rLcxz$LHgqN?>07NMdQt zEy?Ce2LBfGa|FvL!S^EH*LkKf>afV{--;j1=YV?TlXhl6xFs=HIOe+Uz+7zv(Spf* z=vQ=S4*0AG1s2ni?$IOo1C4jsK>7hop1`InBGpyOWieu59%w=;M@fiLaGHcQ=F7oZ z5Jb)A&=BM8U_3-3c%tB+=q^S`q5ECE>|_0xwV#JjmAv&+OB_#$7EwE=A_$9(1P{>- z*oC$}%I#7bx!^Ah`7lIfELgE#gR*UaYw7caa$b${^cU$5edrR$tfw z!VT!>fwKnh*uEUokNOdRDmN}k3Z0KmK^k?ri~UG?MW*Vpr9~ZhYkHStScar2-BKxt z=zZyT3=sN_GFD>4Bf~Vf8920%H1ON7Q&aJ|*JQVCc>)n>kGs?UY@-;%7Y!?593xhh zJ!|n++gQ3b$SK~9`)eih8-`*kJZlWPHu4v3Ch=3-Jql#~QAutRplFzl0?}(_4e3H! zq&-RjAw_kiwQo9&U^4fkiRn5)O!+i_Fi*}Mj|3W~1#>grYM90`uub7&Y?(wa$Em+y zQ%OKnIe_J~n;Q=0ve{|O_ge<*6^Gy>XsL&FfJMZQT{O+BQAC0_TcvtSuN#EBQ&VLX z7W%6R7#tS)NHqWXsDM>=aK5*EbQ7SU6F9Q54#F$gDBi}9%JImp{Jus&R^&J3Xo!QI zTM_}8KIji>SPt%X2U%0GYpL00?oj1v1gs;#U$Bxd0^*y=l%3O+9MgLr<35%p!8ids zQZFV900bn z)G2yG$GCaiF~cIn9J11)DjS|``mfP<3Kdm;p5~jY#;jXMry^z65unpy`s$F~WT@!T z7o?V!dB*W+6p~Zejs%aP0*r~`05o4wfwsu-Bh#o-Nto05D&2k~8n`oWW2={D80@YQ zLO?75-c52sM-92+BSW5%l?Lg)hxovdjWJmKPCsv^ox=7m)jgjh0WAwE3 z2ecf<9Fs~EPZyTwBZs}{-x_kb$=!hZzZao453y>wW&<|N6rE;2(oQ(Sx{qb?N%-;(=cpUjH`gJ%gijyyg>Q- zYZUHpq|!O5rfo*BK8Cw%G*>Nx5u~}%Vfq;x=q9`5Lhh7ZOr=+m>9g$6vgZb@Hw`?( zpzt(yC|!Q@7yoebG*~lM8s_1-Ecx&rOx+VGHVfbJEW#B29}COR0YK-AM+{ zs9XNgna8`~trp3k3&f*h3ujK0*#UQM?U=JXLlfrarJw$pN1@b!`+gR?<})JgZH~1g zEHE~`REl97YdFU0Z1ECu&1!sY+eLm;9@@*og;iEO=N+xC`L+#A3M1{YX0>6NpVIhq z(?xiGJbTpE6be9EtuxCazhQ)K14p4xPJ2g<=f%m-SS*=G?Qlrvwj+q?=Zd7EQ#t5P zr>pD7Kb%4UF+{k%UH5WFk2xY+ysc$Oh+rzMLP{pe2W$-~dBJqpE;bIQRkdVJP@ZeL|MaxeL~3w*_nmK6%!oS_)4;#8fP;#2S&EW7l?rD~4b zgq5QxqVX57utbD47Fp^B(eCu%;PgPogApBC5Gx2u(T-9EFESc4t7rBUyXr)nHU^6A z^d{fV_%}=9p*7N4q+}LPKlio3n=_CowIX^llmQfla#lun9v&xLZwX-CreIbp)@Ikn z#92hWweON~pwo?}t?->?LRW47+N9uNRqATz#aj zhiL}5g#o2L*_e?fQqual!{O(9>;bEu_R-;J<%1Oeu%|LOc{M=kNXGn+uMK;c%A`fy zK{^oKb349*kdvc_mCZyWOJZdJum7Vlz>|9#dcJ-MxUs+&e;nB>-s$R{kqR=%Y0!CX zF=hF|tI%zaMa4{m3J5SzpZB8F#3o7tf@XlZm%GK_3l%M2ME~cFf6{<2Ah6Pg;lzH9 zh!jZb_Q?A#9l_V{KI4V)I@D64dhjf7B$8uz!~Yh(KRfvGeM$&eUtG$a)0)0|FQlU< z>M!&&oqA`Gkr21i%#f^d!Di)&c%F^Av(XZAH2ZucTPyKqHT`zjBt{Gd_!#+cnjWyu z^GQ?YG(Z8M`VKMe(8Whu9>3$PsLMD}Q%^~bj0v;<17_o7^M<5B`-HlsP z;~}RpfXGm2g17@IBwYAF(MdFsT3?{4t1;F=GdUcLo8m^THeHlXygm2K(M_jO1FNSg z3Bw)nV)KG}t-<(cxjiRQlURZ+@T!r4 zEd;=_vnu;YC+Ru*uHPm7B?v9HM~3q1k~A;wSiP4hxTD-MQZGdQ*Y<5JO3h->=suIW zZJ+HSeg{m`$D*G)C^bHBg{d0|Apfny^*-hpPE+V_(;wImJAu}D%B=$ z4)tm0EzosAd@c-D`bliFf%zWq?Iv`ar-;OoNEGSl8$5EkP=j%t#C3!5fo>ezTFVa% ze_kXndh1!}K3n|Zu*W`59m#)E7c zd$n&b1Q6>r%OD|*=6y{U0{U z3So%l6|`Lf0){~oxkj%2aBfoM39OPswC)YhQzac}peGfXD{f;eC`f4Rp+kaLK2-}n z&^!EQvh+=c{{|zorJI$~r?O@V)WSQ~&0YnC8l1o7o{8+x$Y*sQJ&;85`bzSdhs1lS!A(p@q)XN{35d7suaLBgi z0{PiLO2`U3%}YzV^4JF)j9I&7&MxRZFPnfQk_j^s_q-S%%-qW=fkw|&@nl&wb_EmO zSyS)Rxa3?nML;;*sPofF+C_Q7@GlCPe<6pfBxo}nnmvBFDIJ9khEA+x;f)4&hm~Cx zle+63P!q)r*W<=(Df#16Z>R%cg5igxAa$|;WlbnO?V{71czK(*oKGT? zb?<$4nT!d$LW&Q-fraju$Ju{^UW)iz@%Zt)h}sOaAw6(o4=m(BmU8$M!iQ}u!rQ{% zEXKEjoBf?6b^60iIR>sj!BUoS$Y1gxPH&~p)Gx+NDa*$FJ%sk>+Mdh`z_XW`wv@J% zh$ri!B+460uFQ@=@i0yiyUTd z*GhqK?Lx?|3(Wg?!H*;drO_pOzGz8$-Uow85@m~&M9U;X-SVr^71)6>s6PADvoZ_; znx`jdDuGLK05<2YBPaxTPil^9eU`M@?ph|I+h^m9|Hx!On;#vJ&^D!^@b2ZlpE9cU zX@$Fh>3o64Pmp{T5gATlz!OO*3 zpv}h6jlEuTdy6tDN|prcF1QR-l|0q-ETGt>!}8bq!CT6s~k@Y{}AL&sG)DHGwjr<@qV@G+c%isj>g zl0`Vv(P&L&mZNj$4{veHVhD64vLbs+)E z8vzshdL#9qVs(Q`Y+K=^#rgsKd^-TUC&K6!ckXjWsxUUpl)6akbF2-tLP7Yl4|ggs z!Um-}1rW4sA59k14+wwpBY)*!rI>_i@hal8;6)JiccbV?hZ+7!F{k)IHGT!2;ioU> z4o(2ZF^6|+Pp(%u8^k6@a`UnNDfZC|;1|b~2;74jBhm2k zaU<&F>14%b%^qHl#Q$lFXCH)fEx8uxW>!cADwH}4w3JmL5!inoD;NP@>>Ju#A^}~N z>X6UY#FvCo(N3t$Wg!c1bBC}=N8f4lna27tTMJQda!1Iyjc!NvhCltTjc~cAU1T+IYIM zbvCK?e)P>&i|Zpk{5V#?6|WD~usK|UA0bEMH|Looc^XUclRJLNq$Oyi9$w?2z{$-y za>gU!azFoZiDH33iGAIP(J!GF5_*CPzm`})h<6UlehZ#|U|-s(dJ@EZ`XVg9(8Oyz&wNsr;U zp1m~dckc};NADwLH5Zl(Mi(&lfJo=8?Z1bQ0}6vz_M{Q(S>ide6w7Pf_nfs_ zoY<4^Vp-lA`rv^=IJpRGTNmqm9z$+Cpj;CefI_ULwJ}6sipZ}O2a*2N`5Fm%5G}>jax?d=n!*M z8wGBr#r;7qh;{94m=5D%9S6v|bI#f)m$=s1;Ih;;HB=YJz4JSN$lz3qUWz|T2qm?p zv(H{e`-rmSetfb;`{IZ6n0B$A4AVrT{NnYOWq}3I1C@Xs5~fkgbJRxc(6FR zZL{TSKv%WQY&!mRWVmDQm$xYc<}Z=2VwB5Edv}b=_Q3I2@Lqt*jfULB539IsVLN>R z<;9Pd8c<1bo$EWZy?IT9sJA1@G3CIx7+HDyl6AjowylNfaFXCCf}lD5SEf2a=dm0Y zk699ru=aR!5MJ$E$s4QWb3upts zaeVBA#dT%jM^od#_-+6@lD=@yL}>LamgYclq}QDp2wgjELS1VZyIofHTpEb*B z8@vCMhzmu-Dv^zbQz?V>>?RX0eae_2`n(g7?)24~S*rg&aS9|r-w`4m2fpvB~3E;>5&W~D8VB_a?dk|94 zrm58ni*q*!_QRNmPRx`KzjgC}MzJ7cSEb^HG6^8Qwj3k(()*S)Z26RIfgziE5%!Qd z4PAf2Jg7JPb~e%VF}n+zD)2MDsn=Dq>SM?UkDv1am|?a@xc@$fRN_em7;v^1KeOJU zoy%nJxQsxP2)3!9Hyl%pyTT`bYm}l1vdDfw-^LSzc;ZcV$sSRdk-kQbfN(CVv&4?t zfpo4dTBOO%pBb2-@Z;X~_ID`3lx_qrk7*_Uqiu{GR3C;u+uPfv0?(Yl$MF>ke0WVn zD8JS(KkG@Yo%zj=G(jBd0LU{J${b!d1hFWu0Pz9R%o8$Qv!1!clY?$MTJGbgf_OnQ z6f>vkb1yv(-!9y)8)X~iU(Y~YJ&2?2unu6?PxI9YEd>AVPT@Hc+_f>TWoSGPrYq%m znB<|z)O({O@3!0A7+oDlm!}}JGAnf=29XAKZ4OYIuV~a2EeGhhH}&G0cspQ)LOXHY zytd(7F@qHj*>{pKZZ@%4{>hHE3#?0z7kKSRF=y*4BMPs>V=sBhUhn(``9V$VL8VXC zpmRTvre^{VTchQqJW8Bx4^new@ zgCf+-%g$LlW}3Cb{pX;^E5P=5C5sF!F@5bl_s24~X@Et`Rm{IDS*0=|$ zm?o2efkA+TswlgdVw~-BP+z&oG5@C{wJZYx^}ELj7_IdypN5Ax2;fW|Lw4xWw^o&F ze!ASV{NimfWRp$}oT2av68E?ApHej?fg@bac2Y0+94ep9TG)~Ngq;C;IgJ(3Ew6(7 z4p0Ao34-#JZ&>MU^y2Q;D@U=cg^-xAWzqibQh)?_p;_^;7xtdo3U3zVkIsqf-3?9S zSY;~}`*$hw8L=s2Nw#LCrdDSYg>&@KQ=K(xQlMe08OLjs8uUnrt;dU-(J2^ljm2ZJc~G9N#h zF9?ScHdQ|tRV1VjyEmUHY*!ETrMfKIlFAQd^TloaxpKFl#iw{P-y`Oir>rs2mRW|J zlerH(n3uy`wsMQtMIQ|>>-!-t=6=i@ANtYK1)c__f#-p@10*)u6G}D22%E!7aG9j6 zLUhr}TNE86l+sx1VcNz%BD{$lG}a2Zivyeivl8tr9a-qdoM+d2od;Cb$?2^12DEZ6 z_tS<`>Vru6N0gyBz_{!^*`JI256fcm@#`gzkcbf9&|k#Gn(Ly6Jw_C=7Ssala%?l9 zRA{3A-f!kaeLT)q+9lwJi8?0OJj%jQP^B-tc~Ggiu|hYqPA(ZOX_OBn_Cbu$jnx{T z_@flgTb~VBLR8zx6@%M+n3hgk)yAig+a#!S2ENu%rPUK%K&*A~tS8!|r_+-coEb@P zhe4o{H~a-fwnQG%OuC>>`pDWF9fyZH0Vn(V58uceRK!EzKG7aM3TYY^KpbWWYUQq- zHSvB$l-H4)!OSWpLMuD-VNJ*~wzw{mE4{aCIXaXW6Q1e7L*c=M&GItdP4xrlXy3G# zGF*ucvX}2a4I>==3ldyt^gCQa!^LmR@PzxKi!x^iTVAaN1lQiw1RrG0J{PEDD$HZq zCaG_MlXJ~dAds#&u(MF^UqRaB-VLBu>%L!>h${H#0^n`3ALZ!eRlu&cDM*d?2+}Ow zQZ<)RB;%&;PQ#+N(-~)yTc|syPWgH3Lum9SW)o2-LAMCQN{^>iMHj@?GRmwks99hw z5qD&=gcF3FVlTZ5Qp7NF8z$CY=meG5!2U@GJgdNJ;c?i97oglvSX{+gYW3j-C2M|4 zZ&nnQN=67e-#RU6&M2r!dUuJ;mQ9Sr%hs&=ed81Au=VtOlcG|wE9vaIwhRAJ6#Z_( zPQCK$A;PHMa$_#SNK{HOvCjPTR&H}I=OE=LA&@bdo6@$2{Hw6zub_?~xEub#@2{?P z$=E*gOzV~7l|-EBj$Si5OzP0-=n5Ey5;hl~NcuJsx>9xntAAI8hQe}0YA`epC36`lK6MUkb@ z-`2D~M8*BNBv*y#FrkD&(pPRO5j1^sw>PRcOIExhCM|+R!xslrUb^wrFsyahR7*@Y zu+^NXq5{t$;Niiawjy`y1WC8_1Z^Z16Ph?FM5z$T$2> ziFS!-K0V2WqVQ$a7fMAMbKbM=tT9PwuB*&ttnXWEWp~go-vr0A@U_T}_nDwyPjC6?imPn-@=J4KV6gFMpr9{Q7syk2%zISsQSfFAl-`Z_`t zp^U@B$tH)o%l1pym{5>cxF14^*_pdJm&Z!VG z_R4dzWEf^3>L4vdRn&AD89^$~;t#yVkSDelJifV79!@@%9B67F&t6%4S-!+E^F`nUg4N)naz@>`N7lBr%@MtfjpFaIAxNsG|2uIn=Ud+;j^Jq9T zRmDr6;_zy4GCfUhd5`sOt>)!GGK@kkW#^`&fm1bB5tx*BC8iBK)Gd$)c(qpVk;~t2 zy@xaSMM>nZ>j7pwo=8%oz3kprS0cEi%+ry`fab{eLyE5)O~uIZT1AK z3YTn19Cw@RGf#&V+DpL-O+u6}Y5`2X#akc46@9ak9S5Z;M_t<0#=)u9zvEmOgRWlc zuI#*unTYin;_<7%wM`nGKA1t+nzs^|T^rEr*CpT$#Lq2<)51DQ z#ykZ6_<_ex<0#Sk8q@M=D+!z3bJ;$v?YuutKcOXcni_=PZVdHBiPG7x@j9VxX-+ zNepg5NL>oof;GfqgNbqxKi4znmvwUfgDu{y-WJt0e={*;i!%>DOm_aQ5Wlgy_t@1e zdFyIfJ$!Mvqub8*Od$p}44T_KZ9)iZ=1V0^jG>sYeDHoY2BqZOV5n%p0UPH?cI~XL zqFv)$nC}zC($Vx;yBcrO{F_Zs<|SkW0@vX8bWwWbWjd#plZj}Ym!sYX_%34uweEz} z9m1)==XHi$I>$e3E`n+l;MMFdMX0r0d1bM#j2Rnq&j$(Z*H~Sod0e%3Xz^_p#K=bZW7IjQ+#CGa*(82U(AgpZ`)fmjM5{G3UXcjSS^` zO>o%ehKwhu`DyHQF&62s--a#6HdR75iN$%~m@P-xAU`1-1$6D=vZMhgl= z#{elarx=cFF7s~uO@S!iiz~PF0-<7hAVSE#>+*e`@Bws}7xRF@PwAshE<4B_#>E(t zERLtQ!K`AZ->0ODyWKFNlNV-K3WX69FC>e1toGF(Ha|^}>h#_`aLk`?2DEYod%-Mh zynRy$5U`K(A2IU4CcvhR_a@I#%F#O^X%$N&8Y6E+`}ocfbadY_Ti=jQ;K-LQpGzGu zmC@iLbp@58zEk#xE@`WrF{_+_L?On*E_~%xBnnm8zG>yEg2U0MX(sQCCG~6+mAl6D zHT5+R3m*V3FD?t_lCn5Nc7gvswuC4&;H4EeKy$>52cCm@*BpXp6y+3K4@f5dfnZ6F{oZ%gy0_jEeA z`S4AoLx&!AiHkQ&9=y@~N}`P(8phx%?_S`c_(nV?mzvZlFw}%`*7fldsDE;CGy_qY z^~Nc?$Xyt_?B$9Mec;4CBtwT_KU z>-|VO;)xfAo(?8Fsj(<~?awCRU}rs!=TwEFoL=T8znm=}5fpr`4et=>&-@k^`NK%> zK7?B0@3itfLe+Vy34dnPePJHENyD2Kq@?oqoj;Vwh=zI|9W2m ztaY3kXXGD@1f5X^fk3xJx&4`fw(uf^%DaB5eSIY-9aXI%1(SPUy0yWubb(if*}u|} z$h?-@y;Z5jQ)1>Rc6V~5pS#gkU#N`^QRS2r{^G zFUiCzkA%#A#)8$T9YJb)`4>JaccPUH+ybVV2qVJ-o@s5p+e!OMGIo_5VW`7$SZ$)t zX^X8Y^ta#+CIrmD!R-T_Tgi@>XV?FUc6B(AWkrMqYiaPuM%Q;g3|yoi*>sE}T03{4 z!89m}AeWggt;+9vn4#+C6}#80*iF)UP2p|6`^%UD-5gch^B_$#wtUzI?5kDriNQiw zOIRb)^N^IXbBE=RjMPHK>B;8aT{bS&1CCm#i%d$rCdq*T;*XO{qW)!>l7OZIKjKvV zf4K#+6LMM%;BiRJKSH2dt*G`v7{jU}&mBhAy+`yBWGB=^|M;|RHdN#2TFMM7B zuWvAurSH=!Kdf@ zrR=J)ca&=*Rp7IOtc~}4Ju}9nX;gFoeZh`;C&nPC9WHO-rJPU8`X1dptmb)>g>&In z`>s=o!4@0^ON2leFgXhVGCk18d(V5?c@u17WW=DJK;hr{KsHGPQfkEc?!R@ats%HWD z!YiJC=Pu9d|0F@0c{G5J_9L+h=&kH=M~XUH4xOT2wj|8GB|ljht(~J{-2_^1PghHi zDz{u}XDPyIR=t8wltW=M_CAK_NJpz6IVQycG$x7orO8K z?kj7l8Eud`W6Aoo11-UhzygVk*M!bUG&I%$a?pKLmE$n)HB5L!@J4eM18`B87dce6 zV0}ib)v`UU4GQ}2c3~${X}o}4VFl}yvQfa&wsZ(r1bi(rQa&w+jBF3-?x(@fM6pux z7JVv`rtz0brBN32b+pl058^d3L6Z36rh_hQFUm#ma?xH{GMOH&(w^0Qp%2Q?F6oo3 zm3(C?IHc)|tnDc3taZ=Ie2&E-W=L`rQh#cIyvJKN)ONIi5T&P(D~4IY1V>{C9gqc1 zZ2!h^#d4=jKxZUOA$i-H_D>JB={@M?2uHK`pUe4bY}Gsb&YY!JR*GCH%{& z7u}2g-VzgtDKK~`fLWa-DgB5`Hm&t!Bg&qj#0WMe`C{#eCOT=K9p!s~Nw3p`pM<)w ztEeImmtO{nJDbi_*s1~SqH@*IoTh3(KR16g%3*k2u!^vMt!}_fDrImh}|| zZWMQ@VH2Djb`hK%c@8;di9Ai+q!QaS`9x#S6_!YkmP%+7*FgoT>-XQeb=~3xu@10r zNI0cocs{70<`1f`)3&&b(OJJ&LRz6Z;7uGXajnQlm8)icXk9-=#tH-(+o!>6J#8=V z$@QRG&}7E)nePXso#a zZ0bl9;CP`3z55LFYaBOCj;kaaV?75~L-Yb|b9F%t@xK!3$9|*YAg{Z9oFHKV7u9oA zsNqlLV>|yxHsSn=(+&{vLiu?YJP=?pYrxQ#pJN>cb>pn3qW_;iAF5bPm~5R4Rm*kzQ=%%4WWu{0X05(9Y0C7-;w=r!hd`)2)g)R)d}g*;rb+cJOXE z?l*Fq+U9K^cNQB2SIw+G#Q`79vtCWT`DR%WClGOcZIzelDbusBQK z8rJ}1V5QX{ZeP_Lq1rI#rU19X{ep@5?9ZJDRFpWt>FW2M|6*1juw}^LA(i^k)Ud%d zEaI(QnsDkA4FvA|Bx?KSa_qkaH%vdr8HjpA6Bj$rUyR-^hR( z8U(I@{cN0O5!Kd|h`TsB1fgW5Um)Q|iSw&oYaxX3#S^6kOP5t-)jmSa;%qh*X8xL& z(wcM%jJVr!cChyw+m;*BswSTM^=kqBm}-efpT%I_R~O}8rziLXiq1`I@G-zCZP4`o zQlU^Shvi)ng{+%ZG*phnN+o`on%9vXIgQNB&K+1;RX8aqUQMlYoc*G;qd=@GFT9*6hprz(gb2CYbu-b;|MneDxmCb-g*|2X(`$C7NQHP~ z@sU1Xsa$f*^JZ6j&7JGfNBk^&m_F)h9U~NqpvFQ#H5cC^NXolbthPwrwJLo&eSt?x z-}4FK6%R=0odZl1C5| zrYAy_^U+D|R+3>@Hgs}Id-jq2ea?savyA<0GU<;E$gswjNqJ*f*;(o+P*e9X zhQ^?MnYt7GHvn4mb6ITAIlrH$MI~Y}=DwoCrAujno`&tAKkLIb;c9}SL!;?;wsUdS z%;@cP+I=60bL^7lU%<6p#u=@SGu7NxLFp(mkR>Mw?^(|nyqoBUH-R`htJrxGh%Q%CS6adW!#+{D-rLEHMcOZ zT0c4^bOhR76SP?9y4XcK%C{M`W2E)ELrrPAS<3a8&Zq?t1-@NVqNXR}66NNF`#<3X2+k$jl1+ zItdR@^FNDob||0w&U%7!evzts>}h!4O(&;f(MVRNIr;gztmH8z0^INA&~E-Gd2sCs zGSA1d(M6v8PI%?C#GFW6GkgXgv>-2YG_RPu;0!Lii6XGfcO=cLdRzA1vijp54hKqc zlv)~eFo=xy*K${eFoY{c z4D>;HETFC&dc(AJ3!84#pHti{t{$kcf=yY!gT!=*PzAEQQjWOviVOjOH!k#M0Ea&g zS=`e|7EbCKLxD(kB$AW|KVR`e5+E|BdP%Rkz7!#-w5vOmJmEW7_vGvdd_K zU>L2gPD6XEX=LcojCA`MD3l;Gq;$Z34(ZSfKAjoJ$Uo96r9=P3OWl0zt!Ie|~h$uP(uTB2e}W zX~}JNm%?d+QGMJX3Myg7Ze&8TjlFVf<(*^%-cj3Bk@nTk6wTNnmdEXUIn_028_)nJ z^!E|Mbvr?N;s_MXd-iM95c>tqVM~N4-u0dn+N-l|K+XR(#K9^I!Rw9J+9v+hro*ce z{*XC|XiFt?omx*f#FX)mjUFp`SoKz}lv(T9LxQzGe+qtR(BU61E`gahssveIiJjF7 z?-)~|`QF&woVM&pM{WRC&9y|sh}I6XKz3Y3B_=2f9^f%fp220alj@K=0A*yxI;ESJrc5(}j3@*m|C z7-HJCN}*c|Y%h*6Vl&a--$9l_gQ_js(<7+=k)`b9|7pkeb)pmlp_yM1oBo`Ph}gU$ z%LGr!+a(+20n7l;x>0r6A1vuJt{P$VJPtbWICEG;GimXYf#Ip^eYeNsu8C{1nA!4w z$V8!fY44ifp8K33_A}>guB#BgWA2Ic=47??y!!{6Fv|RHXxcTT;wSgdF{w&$C24E1 z?ZV{V0jBC)CKL|LBhSXXrc6?;M@+d6)p7jzD(&Fl&Jrpk2d@1KFLhQ7 z=qzG6X;JeWsM`A~u6nVR`h3}cmWiXXt|DMY8*C9jtYTo`6Zy-xd{Fz`HU=DnmIbRG zbq0qn8gmo~(>YKw^J%c=Bm>1T7hZYKnRAY_eY7Vw7B{)g1SGbjwL}k z_zW9q3tTfm=%UFuzh;%|Y0uz=9 za7*|zCU~`+1`Vs>p=jHW`;>&mZX`4oTm`>vg7Di&TD*~0w?fi@5>)FT3~Y&u@o!3= zX#O$jSBBLBo^dhiqDLV#d&dm{X4R}s>EBMbq<)3fVplgP?wM_%1S1$H^wZ~gkXrW< z)P)+ar>i!9FH$NU7C(m8*u+0I^h^;ZPAj?b4f$91%so_Ae2o76k@L*84b6)O991(+xczbTQ_zX*XK04msu$2f91fJP=M>Og`nDy8ASK{ z$?%F=?euCHjZa0O8l%T|+nOtYp+z~zUVDo?)of%;+3@2oyhCvc6`{B1C;Q}kxDDw^ ztBw9TLgXd8Wft{#koiPc|9FlrY8s7i#!2m5WS71QTQO!jySIuz?8pgl@`r>U74#R1 z*YZZTy}y%~dji!!OBV8GkO_&ZMEt8|*hXY5Ra2HGtXG4=U4`>D*Wf$>PHuBj;QXxg zJKv2-_3LE!@5q_?dFZRC;gnXRIzmV3Mq^q6oCr24dS1WGS2DTXsce7DQRr76CaD@E z5rlW7tfmMbh&4Dz=N=?FKL06TDc(&-08ZpfF#zaK(iN8czTTsliM=3L%3sisRy9Rm zOi8QWwhm4aFP`@ zxX~NmjDlMug+&4=Nb6p*R~Bp7a`&(OB{>XF+YK z_>*W~uN%tfu_?SKT6C5+ex{&MG*~O$p=Ld- zYt#SX99WM@dBX8m7)w>1<(yB#^`&UEY3X^$8jBU=;qY^m*#_K?Mkav%X$oq<8+Evd zJ>ln8sArhJK7P?jXz!X}HQ+B>n^hO-pQ@g3rq_rm-783ol&Rr$1Cr*Pt?_F+4{kuI0(A#RjWp}yNZ^K@2JpT^-!Pw z#^l{*gTXcHZ*f5^&;>LwhAYz;74NI5*ED#2Oyw%@o)=C&B083;6bkmM96`Pa!}^{T zdUr3!bSr*Z^D8rVo`elPka%}-rgT*GKET)rjT%%qT!%yx9aj*34*uBZbsrzZ%x1)ejiPjN3^1V6CM{eS!4E8sG@{TRo}WLOY|r_dc!RC`c~n87zsglg)b1^5Zm2E6 zY97rMPckG7_bOOra!99z-R4qZ0dwu^q+{%H%wKE*(9lRJKJ~~*ooAL|DF$p&A2EI# zp&_5djm#h8#A)ZU(KV4qoR)v2Y+HCvp`4^?w>t6he7H`Q6bl?bV2O*eZn}GPGP(3g zV{1FgN?6jkPsM5(!Y=#3VoZTA8V_8Uv#ct+?hgWXB|H#)Zy|1qy*q6mO~aT<%lT%s;V0c|D~;r4G0C%s6s%ODg?VQ(*k6h1-O{))TtqTrCG377~H zln%_w*=D(aGikNf1&roaK3Vs=7^$Df1^i74WLK6=J9wqLig(~R0o1JRCT=mF!7nXT z(E+FYWNdkc3hFj>>dQi=y#0Q*5b5jly0mboaQ)cAjs={6o{IhT9Fh%2S?c%aviKpXv+ya`` zI&av#IKfsCGx=TO+}wR0>*tRsb}i+ghy*SdJ5)|GVNVGk zR(KHbgKgk`s8wk1s0jL3_F|)|bSyP}qXI^fZe#*(LhseOUga`widh-%MFP3jw92Y1 z$OR@hIcI9R&Dd(vn?PGEtn3@Y2>{uMRJFvMePRxL4%eBlzcCzpgz~lJ>XvV}(Hd2M z!&df#$V#rO1`@0)^-D4*J~QZBS+~-$F=4P;bb~7ybR4~g2-cUK`=A2o!}Opv=jB1l zCp4lv9!}Z(EW)^JZ$gJSZN$)Q}K?H9lgl_w*1f*GHNo!?4f2 zatFlHRlQTi#i#yMWG@{UoYHJLjDk0jMu3Rp*ZaBe+2d)5^#fKWBpsM3B|fA74VS~d zxEB|t|4bWd7elajS9{$Tn4AfGSIz^DU?pG+?6KzfpHe}m>jNkX0(&p{Cc<|g?3@l= zx$Y;#a=NV)K<|_i!Y(c-A`_SZRYQ8XHU@JziJTzoOOl9*FE(>U;bEVmM`1)h!z0tb zZGbEpo-Gi2P>2RG(dceAZ6peW7bU|6A)G0K2V2-7Am%>He`zr64src<>))DWNpBbw zr~{w}u54Bp_rs&P6abbsK8eKN(gV|$-N9|=p*vFbge*C5xyARnD={_pL!riLm)SF>$p0-kp?dG&ekrn|zL>N|t*3cup6 zq$DYB1h+(v(4pZ+ihcOV4)z|mX649T&0;)ShG@K2iQ-(FYfG-`m1Nvx;J0tgTr{$b z1OSe+(_PcgJx?>LzsxvTekrq*y4$(G4Y@P)yC7z~`1($%u~otz>T2$;m?K|H#>pZD z#N$yH|LxA^Nmz)^1x!U^i!2{}actXR(8!d&`YDfPs-6-aFI5~daE1xmX|rl%_L{H1 z=Wdy2_LQE9;EIc?OV{`wM^wA%E0#q zaV;wkH9TO5YC&1Z5>9%WJ|p@^6JC#8_;obog5@|NwU;rSvk#Xua4DD);RmLq{up$J z^700GO7H3!bo^Fy9?1DCr6i?95sj0g$nI)K4El<>#_%>Bslbh_UT{!Vv^B$-fucGr zzwmz>_%=?!?>7r$B10=6uVy$+@TU7eaN8TVEkAFdQM*)~;sC7dQX;f3P*dYKcEgi`FZd+j4_ zy?grFdUr`>O?RXFlG77#lc^oQ zD&Y`DFUDDJZctCPVayt>|96|!PKpO&Y4olZ`yg>6eyHu!ZDICsQFytR!vZxvl!}Mb6kZ{t%^hH=)KP0+InS?WKF=HZu^zDJMrfR5 zi|c%+pOlVKj%358vW7e&6Jvp|ak47Xe$2=&5_5&Sl9gwk?__GCGOrZjL*y>@C&E;>EF(OGREWvVK3$&+{B4ueIlsg0mf>>Zrcw(AK zmcB79wH``A&5b+!=xavvHli%|F}MaW)Oswd;k(C&p|3&3!}*(1363zj3VWT}tbs&d zw#Ix~OPpeGTs)bzI3eevA>d0!$b-qe_gEsndkbYpUQ%JClRwk@eX&?f@@h%=a;LJg z`^&%4Vm2_1%%c`35F{~Z!0#~|0IE(UWe=XB)*)LlX5%m}5oS(v?zK%)HqbSsqB1}R zDl7GvDv!F8jLCRQzKBLjQ*^w;ZuwJC#M=K=SHW;eN+CF5P8HBhB^{*M(Nid(6T0+>iRn0(gL!i^#EJmQZdlRm&!3<{Y2nZ^XdvzR>cbZe7=GL7*5>BKi1jB4BsVb$zA}+=h2;0sf{!@!Czyj7Ruh9oXTwGpn%5>z z{?-zK2-(>q&hJLfgDQqtW&y(=Hma1pv8GD~=s6rTHP{bF(5SFzLroeOHft$5T=!dapO*e4cfZXe?eL#A2t_}Edgfz27MAX{6b82wfMOG z=!|UlUTzh9;+McUwk2)dtBZA*zUF#9mXN(G82mID(|5k5_t=#FYpS}&!b5nVn%9EB+tjkM(ubSZ^%i+zff_>7U1R(xfsfdx zQoVu;w|GkibaoTzbgouK9f10?#B(U&-Dy~=&!(XXnG}-gv~R+a5*>-raV76+eY3yp z$Dg`Yx-;$l(7ngMl=Y*TFK5l-88+owjTB(CHl=K$v>g1G#Uqah?kK{8O5xCpZnI@Y zI>d;aNad?NUO>0)FM{I)%h0`Wk83qwe@@>d_I7>`owsy@G6!NWB@EF2D2~|27xeK} zq4)==mA2Zln3__ZCJ_RbYh6KYI6oMY`XB!}c5+4TX7JZY1$hlE<$~Tm*dN?r^~srh zLwzc1&g1LpV4}|$0zUf)-A0B}EE8WOX``fQaAcp!MJ*tR(NS&{xjLI};ZVBut~c@-># zBldzg*U0wlDK|@|pBxxF$wOs_qu6%_Kx$3Y7IXj5Oe6{-otPD0R~f0TnNm=30b;oC zO)hbI^S*M6W|6wiQoPX-7Wg0eZs@7xnpthbt0TuNh74PJfT6jA{ltr77pA^Y1TF|9 zq{-q{i=W`{SMRLJ7Y@J)1oLMNJ;}l37IA7E0l64TGt{3Cr8MzI%W)gTJciC*N-bMWWQTc?h z0y=pv&6a}A_&TU*n!`V-Mlnwts{*?pJCK>$_5@6}s>}!82M6SB*J&?vYOuq2{SwL? zJUmrN&*y{V=2~a%83W0+sbL{dM#bo5@?*-j9p`Nu{_`dS?cg6Bq)G?iqQan<3(FQ! zPQSAzr%Wz>UI%Rqmvk4}Lj*X+S!vYvQQ;+0%Vz*4A&|g0`r=P|MKGz1Wo*+3xX`SV zQAZCgNqZzDjF~+#8-VTET9HkNV2i$}e;p8TKi*gEUxR<9AMVy0~FOMJJxIpw00Ltx| zN6!11u9;KN*w+_2sS$^2q@M)W+ehE1rJ?y!%ytxeVF&k<;bKi#z_1f3)PJpw$k zJ;FPy(CFpCp@+tf51dgnt8Sp}Tmt9u-~(1@kD3$Q`3Fd+0kf4;f>ZlWc6S*G`tw{s6nua_5dPEz>MAG2h60xM@noffM8 z<{`mgN%BzwGZl0IVR_&2A_4cBqV3BiuSXYhG}qHXgpMXJ<||9Z>`3w38e>BGezplh z%^h~L>}#tXTF;-Iw?{#fnVgW8{HZxd+Vnm%&}N`K{ieK>EVU@sG&IqeApgUbjn_2^ z2P*UoQv1~V+RV=$u1FR5VsOONwW51;Vh#f}#}EYXm$ws4FjVBlwsR6ssgr9ZqyKbu zN$+#-q{qR^-(8(;J+)~hfMX84-LjdyiWM*!+Ky{D%Q0~J;wQlw;F_}+le z8aRsuRib*D{tB#KwnQ@^{VifOO|AhXEt|4}KN^F`*_-=oNGeMx+bg2ed&cIKmZPZe zW625rUx_fRhuj{;*D>TO&09fob@k9@E`u%W>r}!MOyCurEQZj5Cacm~=!QqE3ka`4 z{pmIEHjdO5N8bQVWz{_w`bHLl-^WBfi%z?ywq65RwoIR7&~)fQi=(fHOn%Q z@nOk?3pKp|_m7;2uw&BMar~mihHt5&eEs&x9jU$9MOtz>V=)_R+@Tb5j;9DP2chn0e;LM9JIyj;%ZsEx6LK= zsUziN1KLRAPRb$RXM4x?PASv+s!KUp?C&KltXK{$B<`pdVtU%xDEK_opzBjfJw=W7 zU=lxaXoF?UK`H8LLC%g{x=7IAh7>M=eDG-m316hh?6Q@~LlD~G=F7duSOxARd6BX! z*sxRwKo`?}T6KCdY43NaXas+Gwn>6O)Z(D93iEGNU$t)A#W^Z5VBqZh@W16w3_9Sy z-uyX#viC?Ej01MW>Nj6N3&bq0jCbb^7MV>}eKTf;-NZuI33b+IWC*U9!;3sbh8zuh z{tbEABYGntgFHknq+N#*A&ffa(=`^|3$RxM%a3qtVKtW%N}*tE4M+9Zjje#PkKtr5 zv*x`tMm`wLGn=<`HzwQ)J_2-0n{8ad`i#~+C?ek3vU)G4x@hi*Pg^|yIo1VwzIa7A z@k@VQxD*32%!@MtHw?9}DbB>CIdAl=_J=5qW>cd?bI$J_EIF~#_L_Rl*z{gy0qjIV zsgw8XO)V>p`<#ucd)y1HjLK=9Zc3S6TyifS@P_^zHwN}zszLNJxy1Ms0!gXQSzPP{ z@gG0B!$=}LyyVLN+W_<>AiJ!OT)LioB{3FzH6ZMLSnPddZ)1#jU<>9P5Q_PBwQEI0 zZ0F<(c!oGaFif666Pp(!m>7VEonj!~PkHZv0eJfwvt&d2fhe0{)V1NI z4a0(E-F^4fT#qa)_n3MEP&_!G1BkiEOj<2O_2huh$rvoMQS#VKt+&v-<@xZ8S|S(k zje$gNFqhfF69~fE_Rw5r5lsN8rfxj)_c{N8!(Ce$PGAT1Jp#(SzaxGYY{Umr+y5{e zEgJz&a%7-4tBs%_u2enoK=G<=(h$Lotf~*i1{=gQe3dvW&vY?(LnjMSm5HlVlmGLWYYxH<+<3VaOeLp4j)UaUTWRkcZom*f6mi<*y8C<@H=)H&6> zKUd!OBUWYKw_m%48F(Nn{>|<-zvDUR3Z<%=Z>RI#OEG>Y?WCltyRy*R4rCJRC2WWY zAtCXL_XgZCWc+#RBjNlejm|lpfZP-|G2gr&O9iM%IQ6`;?x3p0ngnaQVZo}1m(R-` zV1fchLxj(RFaon9P$P0k*)0fsyj50xASe#6Bb~eM&*9`}>%64K=+$x=YhMf~FjFf6 z1Qyxyzzo2Dxg;uoJik)p5FTAM6~1V(ldUS(%z z^b(+TU78gZZG#g&)pR{1hAXF^wJTN`(maG>wD+vdffV~G;t!+vS_Cj!^H}V_j z0y`a$^BJ+v+2Z3LTdvsq!_DL8!wY-)Y}LJsFBS#_#JjCxLrv5M|J`pqwlx)%=uB5n zU{%n+EILU^-08_t9_r3IQ%5}UC(A#}1&S?K>u=@5*?iKcuA?l0MhY>t3mP=ZacW_O z+)7r2Xo9Y37{y%diZ{}H(I@~EXz-1htZ_$pAOj@5q_H730UDaz8BVT6S@blBa4h(D z!aY(Kx9lNSW`WcvF>|iCP^*<9`)NtPLwH=u-&jO>kvi0IOJF@F8z`~@)E|f#j{M-Fg{k+3-e*)*iX@Rlc z{fIjrc4xP03N{)GEX;D1GP{sZ{RR7crA7$lk8l!2$~no zrrxWoY$i^()QZCEUPiFgrrqzxTa*t?a<=(hy#YIEbd*q>r>mQHmTsFQqdGZ|b-GA# zUh9ecw+|a|Q4G{hs6OUM!7D~B9B|#z+E8e{B1uwLK+KbC2w>>J`AA}5xo~?mE%oa7 zThSDa2fb=C z5h7`z$BC~sCZsJ8&MZ8NhCQHfIAUqV5=QwAgfS+;GjJ4LFMIVgj!?o(A3{wgr>(K> zG2L$Jd}hZq7~Q(gLyJ_;v1c&PQ&_G96^mZvcEAcx*l`;hMZ{eWX(4S1`#|`p5~ot; zqaW5WC*uS?27qvIYfZ za^{~fqMhqr*4ceQRLp81foJj6+BF7f6##1fy{9-3QUVW5grLLi- zPbD+*6#{m=k|4YAQb?FZqLaE`fkBUds*(bFXIP`e!b7f}XGEB^*r^Z46>WG?GTo}$ zEx;!n=%ZFEJvX8fc~AKHC?6aBi3&86r*Y&uCYAKWO+#bR;GgVFDDM<2t1in`=~9fQ z!{;g4hWzIMeFq2a8s!LIy{jFtVKpmRSi$3{(@S9b_-$A&e8SQXEu%;`EHjw&TMhD@ zQC7>p8O1>a=s*(Q?eVEj_pphz4%QzmP@n%>IYa=|35&m401T-F?=9$-V3A+-K2!Uk z6mK6Px%Hr9lgw0;QOxV(dSg3MqVGDis?@LZlNW>iTb#W&yU_VGW93(GO$uo#%~12& zdeG}ot^{KhUNUKzV3-{VX+u+omhGm!`cRdJ@bxgz@|x}o>be8$Mb(7ymj zK)AnmzlVz-UNZKof&Evu9%EC&lQDs^4a+;|)4%}t$FEm{1USX&j8ePzHmHEI8;QC9 z$GLYO{b#MZiDn>VF!E;oK)1w-$aDJn4Z2%VMr{AgsXw^Pt7YW_@MkVW!s~NUS=(yq zjVF`c`#etgb5!s5Q@V64$@y*ZW{Mxq*Ic|au*@;s^**Mg_X0%5gs4VjQLA+;di8#h zH`*rsxhN{{r{ekeD>l`TW$vekadjJkY9SE;LN&UL(Bu2{wr9paKyh(_3n?Ks*)@7F zywtVx#~R-p0K18OVh|HL!|zF#QyWI-L1I|}01#pAvKxEh=01Z@6Ck^tFb8Q)#dZy< z@D7~8On)S}$hYRcyoZ94J|uQ?H*|S`IhCm9;*yJH`wd%3(ZX71Wq_$wXDYrwA}E1l z)V*I>kr=OT!~{Mfw^5aO86DAD6Crsto+D`T@%5AHJyIt9OkO?+Z1J9 z9+--O_)iL8Kp8NP;(Zpt3dkPz5@u7)OEVhVmSGTcI+3zZ+-w28SwM~f#MWi3SwB9m zyhB&kXmAAOD6~o<{J?h>RK_r@c!5e_dmYt?8hEV(3CER+C%%PzC5LB9CaLv)Ck=Ev zbea;qnfNtzS4Bgp&Y#eccmqNB2uZL7f(}gOr@)^|=TU}#n*tY+*rm_S08FLQ?J(!` zUAM;6fL=tp8#JH>)y3`0P#k#q608v)h=yrNS2;1Rhm8iW+!Z%Sxh_7>ste4I@tR^b z8iY}YA^M|Pv1j!so*OT|=TCHl(yD6KlBwr=*(dH#n`waIvvu+?R$lS~vxALv@J3%U zJgZWAA9%W~OYQqVF3Z7BdQBgTBBzoI# z+b~PUyIOU>8nHSee}ej+`+b)E!Q~+ zBpP;MV9(C#O1;fgp!i}Usc`2tG~rw*$c^ zM+vww@Ff|_S$>#@t@G%wk*rGg-nKoM-23oy2eiq1ia*xZ+Oov^Wm*S4|Ll}bp0`Or za-rx=>zPnTf4|dLEw1p<>PUp)h4p5EtoRZ5oc?BkH3~&FRPh>{8rcPW&Mrlr2+JAx}p{+z{4w~=bx61!r}u9WIs8Vyq=yt|@W;i7iQgtru$ zb~b~hrK0etit76RFP5XO7?j_Ob?^cv;(o(?9l7Z@n=r_|esDMOVF^!b{t)|BQrq;v zT!n%9zwn>p;JP!~_YGz6n+)5{7F1E6F}XY*^JTEZ5gdmd%hDFW91v0-wW?cs11eS- zQ>elx2CXY`!3^4$d}DRDMB@>huWDfd88-RYQttl>`tT|?p?i+R+V1*Nc6FTICxWQ` zPPsF?e)a_27DXN7Vl5;z=gO$BuDbvIWeO|yA&?is@FY0r#O0JEmc+;r$$(;c7po?1 zCVHK%VmP!%=O{7J&D$&x3lm4T5yZ2ufMGABTSK&k85Q{BXak!z0et>an zNJHEg;KjjVJe=8%L{1AMvjTaSJ`?)vUfPF-Q3F@PH|8ejs7feDMKmOqjNuXL(Q&bG zg5(nK8j;3MzH;<*R?WMi+DwRnY>9rz^Ua(?zNW!kCM;f?GlSbpAE?%w%^IOLQ3}y^ zOjH>4fmhHIFWII6V8)$nCQ1U283XZlh}_u#=;2N(C*!f7&XP!|0t+M7ZN0I zcEU;{eVPjMf*!KZeKC``5DP%pdvU;V%(#JWLZn4_HZJX6dMcLis$jZlMJ9``P+mIU zN6uMlhzEgg`pj`U|4Fk*ajxk6hko1%wjt|Eos6MK-Ho!Lm4MoTl%S+P7 zsjQ$Bb7muPROtEzgv#9^;}JNYYuW~|;YrT4t!U6eS1fxoE-5?~DsquggI(wFzlf)H z@*zyZ2h8aD0Z9&^jew=^Re95;01EH`kb4u%{Xz7!q+08!VmGo*q9*z73KuS#Egh21 zMfy`k`LaE)TWb$sS!4rk<|c7EUJDg^gq9|4sr9!kH78fC^phmrG0F};!22FqZimP~ zMAyQF$;F;oj$=qfmi(scXnTX3dmy-b+CQI>mj0R$u0dsSo9bz=3?|i)h+~=_dPiab zLMblw?iYe?y+e6xZ6nhej7m9<&8cViY ze&{uAJq5#9Op8vIqYXEm0rY+gbh$$_*Y*36Baxk_a_$B}gN=P?f)M$?MDp7?mLVDy zgI)=SF_iOik3Dd^ve_d0MwND)rSO=Tk&D{8RC3JpHmoA483(Ba18NA)YB78GNaBz4 z6VlrG0_Z|$mH-?7$Y7hy=hu&k$*4hPYdzOjH$W95!*2vL_4aTB7oY0mp zbmvB{KR#p;p~r3q1jUrlDpGi2pQ&>aW_%i5NJAtB^cg76M$9ZBHM-0HY1k9#9Sdbr zK4+@xT{$Et$-T;WniPX8fa6sJ53gXj64pwmzyKP^9P9OyN(i;|f}dHO-x2{K`L40Y zp)#3j(E&mk08l>w3)MH{zG741SoprH562%z7FmLA(`$pw-Y_3C^zJ~`U`fEJ#$)7Y zfpW|971ZZxX=ERulz9zj?zZ|WTn(lVQexZww*0M#%7{Af|&qdi|eOczD0-wgN}RTmvKDY*-{ z(F+OKi@pkLil8Mq$TS%&I2HLlgM_FyZoW+%DAO*jzW`7tgum!=X~- zZp$_cm7HlDhmKK-Ui$OU;?q0r%&qw#s7@rlx<63JJ!(|COk3{d&)7^rH-Arr@l$uc z-dnD9dwyE_AvhF%Rzvt7o?@f*0KE8Vkr@IvW%?2>g14ndjMe?K_a&5-tBc2`f=(GP zB^6h(uv~~1GYpC*Q8ztlvBDuXmlDM9T`f~b>TRBtS~Sd!-(S>lgs`F#U6uC*gr$;k z9S0Mq%XK%%J7XndhYjI$cU`6Njn%-T8>q!XKJ$5M&FB%`K4CN956z@`0}8wX10>a>_nyowAG-ZoA5#8q_$omonVCR6RYAAF8yI6r?k{aCL{_|tICF%U;NfZtRR_cG-cbtnio}9x$N%t|@7Ft2YEBIfG zM;a@;baR@SbDt$<;J#XDIhZDn$Vk%4OG?!MZ^~6t3jm+sm(QLJHaVMEWhFkL|WIzbl&e=1Yp1pZl&u4Gdo_ZkQ-Q*eSXtVJMQc+%zR z#W@Ekl6l23=MZ4P0B5odvhHGWc3l`1@?F0EpIQj<&yF$r@Gh&aBz^;`Pk+jOXCoVY zYvyanc|Mr2!&@QF1o6WLXgFud8FbLRHVW3GL@^~eR@Yx{reTq}ss;D+uZIeDIoBFM z;+Q7!J-R3OPukKx?@n!00(li@a112=f>LS}V?!spDWqNUKt8py;qO`5vJYB5dpJ6? zOEc&S5LPq_2H3n>`k0^Rja16GFutzgeJ}7J6{~%~Fpa)sm>j1@yt-t*!IKgc6VlRZ zgJzOC4k-l8{eK1V__ElnZQ2A7wr&O9wb1ecw8FbH#Py3J2TMQ(v9+&n2qIH)5Vdz> z#_F;}AXI1JKkYvesw%~1(WRXyl{-$FgorvIwaZz&kQ;#Rj8$j1rn02yfGskTejs03 z@iy+8#}HJEJ&_LbkC9-&ja>m)UCPVi_7B?wtvz_FTPvQ?L`}%Y&kg1`u@)hukkQwA zz_Ji`EA)R-twe#C5_VDIqWz!wWm8Z2fZP*T-E`#){&o=P5hPDAm45c5PMt;$35s(p zI;_KnB;E%VBpE%HLOWjm)?)&SBf{^|9H`S|0eY{*|A^h(BMwe)vr~#yx@ku%A8LBW z>3;Rq8!{wT@vVD+ZC?hc>yl?)8tq+qv|Uz61@6K3{;kQ2T9E%e-}Yq`&=S)AMzg!O z%p=IoUh6PD*0u&Di9%20$d6h`mvk(5T&D45&%D=Fy$Q8eI@qP-8Xrb##U30p$iPdg zMf!4QnM=aVM;Dp3ld)YD1lJ3(tuwtaYe?rh7n({>Wl5bvZ80td*~AZ`gg<$W-={8m zRsi_LQrbWCHNx+TLJ37Gu4JL87Hq#dylYngQ1x!1uT3Ne)@>%tX1tEW61qW_je!gu zj~@N6?fkeqx_wR;0G&p=MVAsRul~DuXaBYoP&uA7R=I%{r_jAP*h4~OUE_nHZ7Qku z!VlNa@NE`7HC&yV7?pd|tzfn*@4Bpq!J6AF6aV$mbEv7AY&T|xM?d4J4w4{>d-eLP zZOn$+DL;bhAOs$j6?o`gxlMaXtX&8sYrLV-Lk#gJHqjNw2jq;$DDWtOp7UZoE5+(2 z%)DC<{hNfo@O{%mk&xmLiAj)EH#uSC=Tik8^H;Ll#M~p1_jypiGN!%w+Kgb}C-}(7 z3T5xD;fv#WYsqCYCzjG2ksJer|MMc?qAJb-qcIre00k%EX_bQS#^ygBP@_o66AAUh zWedvww6TH+Y#IT{chArt13?aE4Uokc!DFJCT|Q_fj^qszW9|T{?yw%#h6Vv*OR1yO zY~K!eQLb5uaK^0z=TUQ#9pa$c;Y_Li!i_R z)%sVKa39t*Y&SYCV4yUk?sL};9SH)#Tv4A&Zrc9?$j|*z7=w~AGDyB51~(nz}zLdT*SCX|Lc=0hf8w(_9qQi?RX%p5*;7= zB^O7uWH6C;r0Jm(m&nK$oWwc{JW%dYf@Bq@mlUqdoF7u0my`kX*)G7vZ~xo=o-Zm| z>XN9!xExf#5YwU+7iGBxmCd*&!&2xiGClb#kq?J%p)Wp0a<{EbqVZCUuPYknQ^G#8 ziE}BTcM}(8(v1Fhz^bvQtGZc4cn(w7ua4T9uSryY1-SeJwS1MCu zntPtO3&}a=D9@ZkM+`0F3t&zKfiDP=B6!$m&n)oSJpSDD-z19 z=@E5Q+Xd@!A2_~E#c60wAe>zxlu-ev@ur#~mJ<`)b&+x))d@|_TOUN=%s(_$S&d;N z%=T3REtD{s>JH_R_#5r_X9(+xFrg=O*oJj4!?{F1%Ex+p`Syp3`TBbw+pcU#jt3w# zOApi4E=_hsNY}hD;UTZ4@>oTf%0O$j2FBWY`DoCVmB210PUAEMNC>D;;Dwl)#6_~L zP%>PsN;9y-8?oYUDZVZog{&tEZ4yo~FO~dod#$i^Zfy{|yN#=q5|HbtM;v&h?AV}yE-wu;i7{mQnQ=5s#DQ#p1^?*MKLO9JG#lzYU+n}&h1(3<|hGu@VV~e!0nP( z{Waf|xE>T{(Td@iGbWc~4~5^XevnNTJa#+iofbOMK4LFI88a`JeQ{U;t$BPY2&t&# zPQUF{=MeNGKsH_02G9aY$nBK7VgpHR(Rk?h4m|iYlDh3iWiGj*;2|)l9|kba+6==Z zm^NG-*U)J=ZYp}bJh^OccH0iFGx-O3QT0IH`~9PY95zxVOoVO-9P?^Q_r z+)CY|=?2#)C#(~bve^_imoq#Yr&Vp}G8)J3tT&-{(&Voez)VzpbDdnN7zVzoUf(YY zB`U2aSB&WIBHwRLL+S?74DzCk*In#dJkw3 zr`Begb0A@G&G+rxOq|cNc@mjX%%{8zbo+srkl?hpvhIS|BJ1U?iSV0<#yJtGcHwsK zS`&4h*s@i%Z(OAzM{j((*w4$fF>)Dg)fP1L>o?sl7T$5UCm<8CWNrIPB<0xQwGaq2Vd7N-SIPsAqK-tZ;Ub zghV7?c# zj14|ndnW@JWU6q(+6QOKcj0;2gRP7e^lO^A_Jy{M4fOXZ!X6dj&?_IBP|vp;{1SFC z>+&7VANVa$yPrPJf`i|8mat8ChZFuw8_SqcRgV}I5i9s)oXCjo2wF#c0_*Mo161Te zhdAKm{lk2hMXsGYH?bq1PnN3`3#tm9YGEfYtjsK9l5HKx^LHx14@PV6Tm2qt*Y__J z1axb946P7F0J_VlcD}vLE$Sum9^!UiH z)uW%lJ-Sc66OIT(@(M{)#&XkD@m_`hXOrI{x|n4l&b#bQYEr-# zh}K!&M?q=*kA`(&AUfsqNEKJT;Q4`Fwn!LXV&pnf1fa^4AEw7=`5c(Cif?^8Im*D* zRNH*z=o{_*;^Q;#g8f-K5+V@YDm40H-%&Z~ znftRJ1XQ`KuK+qblLH&!v)pRb>r5EwqK%)beBa!7CizlEikK|1>1dd6hM}Glai1$ArIazuCx92s(RsfqT9YqIVRBXn-}^M zHz7vh6MVpoWiFL^fm|aLnF}nT2hE~ASs?#y`Xz*OA5R(iWT?}dHC~+ee!tSBkx^e} z{`=b9>2n6TVkx4G^SP;ME{TMm3x^qeHD_xvAkvbw%rbH5AAJ$gPma?3 zMnhO_u9cHsbuA`L&1iB=r!oxB0KeI9+zONBM1Hj-r93XsR5^Av&*%E zdmyE*U_saSkmT@QSHzokwscdd#65X|qdX)g&;hY(9!OP=$B4I#aTrZGabx6U|GfCRU}e*gMZ8PhDV%3344 zK!b-5bh;H|SCWQ4no5+Y6UsXuqjt}3Sl*OUFa7+{kIXSyeWZ?KKQd1DQOc<8PukdZ z6E&FFlm6nkJb(lfQM&EuMBh}@(5 zU9_gZI=(9C4hJW4B;xh8zlJD6%)bH8nJ1THIx?I@D93T;&S^bblqzAa&1T56-Obqd^Z!7&(!^z(F1 z?Lyw6vyZ6sbfvY7hVxd5V^h8Je9$`Xi+Oe1-X&T|mg_#Tc(m>|0!o2&L&=yM7!GDd z&YO$Vr$nr*3tkgSJ;O>}{Z3B{VN@7My_;TeKhkaTAf$HB`o+-hVZlHE4=sT`Y?;b! zzJaqf(a5Z3jdTrV6{0$M74ryk=P{6Bia&r)<%!kC9ybuXa^cubFq?K=X9hv>6`%I& z0EXN~u(8j%Wh3yOY6i1Xc7W+f>#-R7UTpb*uNWMmY(C}me`+@<$5h_H8DP? z21+gSXNGpU1E?E=h0YmsVb7tyHA=eyOjD93&6*JizKykH{D*innb24p81P&cR^vqY zf-ZlF%Qm^>ZZ%n>iG1}G5(3{#pI8IUK1HE=ft4{;4lyFC;F_@b5nH5%$!ru~t*6e1 zGjcEq8Fs#tkKpl@C)U1durFBMph-YC9MwH(qCDCBT~g~8SNFf7&(At3b>RJ@8LzHO zE~qR3kb0lzesjB}w=Ra|YoO+(>}as2)YWH3iXi)+WqE2k$nILErgt&rX5XtvC4BD- zm%rjfjy?}~i<9@*9VGiPj)|yLdhH+*dOiG1+F~bzW2T7Qr35xC9>Ohr#_adQRr);G zf767)Sjcpp@jimItF>XCUoKSq@ga+(vd-8du50uJ=LHYLR| z*=NtioRaQwSx%^QK z5Pxfz7tx21iY=uAD0s-g>S3;|(@aJMs0<0woYECWMZsx(pnY!pT?!Ze>DV?}b!(Y! zXLRbOXT)2poE0SLk1&MBo1pzR!dukf5~)i+m+coHk;23y_`xay-p z@OTKF_QmH72^$sgU!KAVRiHCilnT~8k-LxKNWGfr9@~n64q~NFH9-#-?BjJeNVG>B z!$JOA_^QUpY7Bl#(C$fQgHBwaY+>o7W zNFUzK=tAP-G};&2-+tBB!-grfna)!fzA5*Xt+!eVa6b;WF<#R$>nAaEq#^NB0p~!) zm$C6m;&(NTRd~|80Q^c}5e6F6uzbhBMSw`jK1VX$PWDxzEHH)uW1IlnL`s`=91>|> z3J?h8RP*vHz)S$AOu$W5EQ~l`-GWuk$Y6~lSa>?It7I@RD@=FQOP9w>DL=n$PYG#> ztlYw~YV&3AD9ZGJQv7$N@H3pqOOC}7vH(;lksLIfpJ*25Vqa~0;!VqCof&2Ft#dl= z;Iij&3n$|3vZu48I|Xt?O%$IL+uKtQsrhk(pi8#eJ$ZeSD} zYQv+h*O~{Q6=qUt3`~M#PSED{E*S(+mk3E5L(P7V3@5%T!lWo)xXAWHd9vOKdKr-1 z$8+qzPh&c?0hii(%|b?Cb={wB&|vb=jpX7tfq?N=WNp#w3WmtsQ1P?Vpwb(d@?9&D_DL-tm%1GbhIRyZQvh$Z$&C6HfU4P#ne6wE4!9`Q^^(K$T6U~A2q3M%?w)_U?D{xjmCDqASjY?Ec3EGwH?&6B z7in#}@wIzxJ}*)NQ<<3Ea9_1FD?TZnCam)!D$;n{KKv-4Hdf-VC2ZT<)owrYajgDZ zm3`?!f4c6KLkl?58GZ`#f8NkD;y!owvXsEqXef_qcSy1cB7e=vOHzPyLu8V4nIEjH0b>WefynaY*2J|txdsi0F| z{>TGipIC2|EA|mcN93|3I_<+KMjQ%^G}3GSAbEo8$qy$KTu4mI8Z3N_PuWcweu&l> zmgG|1ZA3(9?*dxurfQg(}RRlaH%CxpM{O{WzMy0i2M7NX|jFXJ9%~6UGky~X>p!XnU4Hapj z5FbVrZ%sN%!ibYdhwSyoMt{cMjoeCV#pS+Eso0s5(@py02N3=!wzu*`A6cg|{G-E> zC@4fj4Hw}t3YcyX@IbjWt*ZO!wFu)SXv_}f_M|hJftv|Xgmyl_ZO!+vi{KqU13VT0 zf|o9R&bHQ1?G^>vO6=28J$0RnC+PeJotr|Z|2z@_J!I*2#jzgd2G|wgsHav|35*;* zygyoP`FWF?-&*n6@(|x+GGqwwc;zumi^n3e^HXj9)+A*12)IyPkF=RU)={y6aB)R2 zhGH*u>U8Pmw~FrFxN07bObJj6fayKo9{Ho6Am%>d|WKjWHVKkWly+;rcCpp8l zHli3*_~yEmoo*Y$i<@y#?F zR9_&E+F0>R97fK}Sec&Kp$E}*3#DFQL2y?*Jp%|`Q}t!RYsPgi=xyPPjdN9Sonwp5 z^iSPsle}#x6fWB9zAN3#Cu#}+J7~n{y~Aua`O@6zkYTcgJDtl3ba^PNu1tz8lktQ; zYN`>Bl>aAzu_SkR34Bpf+Yhsv4#^V2sD}038?xmw1o26TG>z4#fB--IH%qvG22Q&^ z(CGX8LaVU8xZ}L2ZDKXj-kp}@nK3i5Gc*mkZNm2yuB33HCbUvYK5@AWax*RvQMIzx zE-<%T2OY*8>g4gNr;(Wo&r0qD!Lsg$l87-ArL!fi&2kzhJ)Se{QBwO4xSy(8W{|m@9 zzx}qq5sx$BMiBUH)_`?$HwI_CM*Sq|GLd_Q(7LJCZr5JRLI5L<7E^qvPDzZzU5g_I zRSYnV=0Ql!n9fmX&*#k0$3->;LiHLTR!5ms-&GeGkkJq{ts+3pkhCp10QesQU&nOlnyJm( zVnXul`O;9(tf;KRYXWG(5d-enqJ7YWwo{{&j!BP^pWFQgYhy5qRg|O_wmGv@-U(eY z-$HJ3R-YQnzxdU@vsDEnpQ>|P8$lw^xe8B5mC22ICK?{21HX@Cj2d)oie$+E0T3JL zQB?KvQ#f9wKfW`Vx<2-wYu%Udu_lK3I@2qSX{U%#q4{L}1rKlYp`Yifo4VIgBBue; zw_UoWvXE~?HxG*_x+fX+>wZFLtUp{XyRyExY&1*y89*Rsp_+RR%-Yn)mz&m|Y8<;n z+LHT>$_HoND_~+z>H9oD!R%2LN$O_lfHNMv5d=Wu%K>Eta!%z)ql*#ThF|S5x86aR zpv%N(bPrx&+Rxv5>%S||685=5s^cOe>H#iOvhF_=D`}A4trDJDKOgQ zoTLN(+NyTQcsaCI!aXQCm?I+eSB^u2m9t+|7O0I<%=J1ZeB*exv;v6%*7yqNy6+-J zAG@jDFk5~_{?l)plU8DczPU9QzJYjji^$_RJ(5(({MSQLF{cR!vW(ckR()+kC}cJU*I zN|=5xk42pBoMO=N=@2ognZhWlKh;t_;Hu=}x6HyC-&))FWqGX_kvh-y0gl&X{CU&V zzKO;|>p&YQn!e0kmHbsf^FhEe1cP$Lqj5meZ?GwED`ke@w_XAEU|#VkBz23q=!iUG z<=UY|YUI8Klg0@6hx**X?$={#zzE<%97GM+dnZxA?j+GX)QW|w$KaV2B{s> zQ(6x?YjZ%nzlYuD9NE&tD*o=eGDP>F$fa)Dk~d$eDY+#?@u&?4_NEevIiK*BhhvoA z_1{qOYR<+NRAjH}E8}JH*tui6%5!jd2TMYje9;88VA^)uia+=cjKYRLcK}FX=U1TG z)v$}dsm6h3VB3sT$Ss?r*3DWP7Uld`54srtj1sZxyo#3<&%gwcbfP^nM;}r-d~(w> zEBfJd|8wK9A>;_x&}!eA1Gwp~+LJRg_Azpor>4$~gCT@?gZAz=HHOzyycJ1p#c>c@ z?dmZG1g-ZpA65NP^tNXH`qrhYDVeZeb2E%LmiF(yc*P}+@oY)=0C|0>eUd6!%F@Av z2>R3ewod~?6lHl4aT|eJA{n~0-cP8|o?1w(EnP6dO0F{P7b$TzzE`Yh<=}i?TiP@y zO+63nB}bNO@PKDf3bRbb@0%+$3Y2f|@|Y3)ZKnV^*W-GPxfBc~53Qa=k$7u7vIvC{tP?pY` ze~oR}0e-mvC!urR3SDdAa9VSVmqhK~ZA$=3+g^F3JueDZR+r&qZr**pPYfwzFwW$< z&2CM2n0i6O+NqI&Aa&PVJ9h0Ic?o{OS&gUukQtBrYOqiKD_DWx2QR8&f621D&`6Wm z!6dL9p`hzcw%0xvJ+Yg{xcfv&etmE^zH#4|10x#^{lJ$Pg%n@WF#4)EB}!2I0NoAV zDU2jk4W$kkyub)_7xE`&o~{)_TVsBj`*dTU9#3!|Mr?Us=2GBB+?|+_QY08{stZwD zscUoB7(?eDU}*{t8@mAYzSZ)U$@*laU){P(xMc?3w%2&J0Cv3Hyfav(NWtut&P~ zv(9RXdAkI?ZXxG+<4$$D-QBPvA(BXh@pf90I1_-7E3orf zc4l*}$_;vA=*#g?F4~0{KG-{N=X=LHqsUdRpK8*iene}sFHcC=8R-tQ#MW|$eBgnf z!b0+}Z6?~L0r(KX@rQG*h+C2X=}T8QVbhp0sOO$;LTCg@oe3c?A(E53-+D>4yAET# zR})(@8P_ZVFN@A%i`iJ6(~TAySBm1x#@hYBAJ8>|5mS6M%b06^sJSChCo1}>X6`^0 zNmx)v0>X=QyO@sptr#E>iOIYwoTDU9`y$ScBxV@+ti? z3B*>dr{cfWAs%~-`+=HM3P``9)-AcYwFW-uN;l}NEQJoe(JS3!!Yldah2>x%()PDn zNUYzod+u2=Vpfgid4|I{2JI$aW0n(1OZU0`S7SVK(_oite)&O=6jaGVs+Ht`h`&e3 zjy^dUgpxf@M1cnJJ#fyp`fmw3)80|^Go?5^>&N6sPV6~FCKzT*jp{GX@$ybEsaLp*k z)y`coRJLCCGtB{04u2MyOf*4AmNNZ)RVa}rAq*nto0Letv4w=?_ALg>vqFK;=J+yB z)fDHU>(0`zlS^F>o`;i8iTb*!0}@CprIskm0|-k=e_7GKKuI|mo(r-=5QNd`q{!e=04NXTU-^xd9c^ps7=rssI_n?D{~ zl=&OD+MPll+q@C-kDXT$M--~uqaUWE1@EhTKQr#c2$6qQ)PYEJTq6X5k{T=$T*8SR{&7mg=+IK`f3+OWsF7gEcg2ESB%vs;QnpSBUQ2dByRXs%W7Ia1gw0 zP&-9pWl(E{+iioyo)}VW?pX1H;E>2^n!) zPH~hjKUH!`Q)ZX!hM!YMjmA;?-}5R!Y$)k0j*|_LqiXJbx^ci1?V$f zEhU@_5XswN^*Cl-J_S%Au;kn(@NVYP>1qlacIWeenv$~1rlrZh*J4W#Al0kIYb}@3 z`6w@r7ZkDl-=Z|(zuGbgO$4FO@xr3+X@Bl?cMi2*246QI?}z5pd?>>Zfm#)<)4$ln zgH8A~Bg^uL?(f0P^mxiv5CJ5cGLbX`qtK0$HX6*uY;H=q+S0F6}4hYn^X*=3XPpq{ko0I%BrxDPO(BI_$Hmf1s2%_89U(iOyb!M znk8B|x_DIA#dmhCh0Kd|iMas=j-W`OAN7B6$T63go~~Y99lNRjF(R2fzELqh#V{aV zlVB9Emdc+Pz7;(r%r9O zhK-N1rjq*`tb!SOXGnGRMzFwc*pMdkVeQKpKCYR~R!mM;J@yE6|8O!5)D=hv5WUAP z;&)`MT05+~-;)TN=O{Ly2VZwvPAW@xoT;C#Hg^`dkIDQG4>dB7 z7Q5kTX#vdwL>H#=YyIImp{6qgb38E2xD=5^qzYzDNQGs!+>F7P?*w1dhr!bH@7!Rs zyu~IbdYR|c;`^cDI}g03&GAdXW+Q3`>O9v9BK9kq$J~1S(Xu5(1!1x}smywT)ZPhU6)^~5NFo#7Ba@=ohX85M z#$?>dSoVO*rDKlf5UEqpbgp$LYA%>UJY-c8_Ar7t#O>g2%~taj9}L9Ho=vK!*~zpd zxht+fW{dFV)BB}G2og$58LfrFDzi4IRIs3gbbChBn3&%xk^tyoY+ z5fyN!#a;zee@fANUIEI%PokY6jzMdhw^Gs;KMD(HZHjN=$&D)SPVZT1+kUd0orOSR zXC6Dog~(7jn+MmTGn&4*Re_FLD7OZo7a!yY35bZvVrbXQqRf9x10Zq@%W7o(5g5aw zg$uc%tgm5y`8=cPrCaKLd}>7wlt%r|C*)|@}=_TE()q_8b-u=!SUAX*(3 z2ZmSd;Cdt+a&~e2gI(;$o5d|ee6kBX6?oE0rmJ2*g_>otxNVHZ_-DG(Hb)qDKkBEt zrr1~BavG!+0LoFFHxqK@_N7_fJbQ~Ze2Uy_VuZ9qn~r`_Lnhh%b7Ok!e86X#cONKQ z7ecbV0YPv$9Vz>#lZMYbydo$pRiBVvPVuPAw0HT;)Io4m7+fZw?=!%}@qQ+jmGZns zC1CyzI&miXbAMlB3j*RZ22bnG@3qZ>|A%g!#*VB;`OPU1M6+U-02z~Qh9vX~0a4#g z<~lKpa5$d^io+T4@&_=vcyFa(M*X)QebF!dFh{HG9!)R5PL~_l_vtZFk%!sCkMnwk zdY9!k($Ukq=?W_CK=_RyK-Y9XFw}Je@|T7H{^68)3RBneW0|@!GgW>4d&ewRgQ$rP z4#3T~g$B?_XGGY)c?Bz@|*g2MzRFs_W?Xyd42 zfkaD+Ir~YiVbOFsm&Y(m!E=_pYtMd8eEE{^I+*WDvhofsz~D_Re8ofWlY?H9(bHst z84W=~9o8j{{A>ssVC+zVLh;PYJ{}mnMU*H60iuql`Dd_aCcgCUuA%#;nk^UgFkl`c z#~t6->>wwv2vnEbr0Fs6IR57pN<;SBAs4dELXr@zdz^uj2RL{a8qsLLGv@DvoFRy| z({)-emSk(4<9*y{a1B~N$7cKs?7b#A%kfT7-{@8YMA+n;lXa*hOKA@x zI@O8@fch^VL~NUY6d@eApz194F!y?ZmcC^op+?!1vQeeHED&%6OH{szI7Bc1?wjQL zM`E_ORva5*5nf88^Ce{h+-!l}&VS=-bm55NpA4oJ&vX>B$u+(f{R0TFHUjarYbN|| z7~mVxbO9CN;iR`Y7(-yF*&|$dniG+lDS14`l;UkK zN6|}8el0x9HCU|2#Qt_jGN)D`R|HG}J(#D#-Yk)!3OOm#ROyaai--j0O^_%W#H)GQ z_MzyNkwXX)V;izCf#~iy#%ab0a-h)SD`3@Mf$VB?N8+}C z!Oq6xu4Wi9p4)W|yt0PG^wM@LZaC`Z^cAJ#3zsZN!6`&KtpUb}_HYRBbQNILGRhQ4 zx|<&|{%@C6oQ4zW2V%H(#?wc=w!%9bz+!BXsF^c&d#E+zTpqb})J+j(00}qx+n2k%LNg@=(vvE=9t!c$Rkz0TvCu1xGV_Iy4#xZZU@CiHbUbmM z-hSeEWV#NaMTMenKj6Dc7deN6aZnEjZBSKBFfn=oYUS8t4~`bN?m=$*Ovx&qtmR?6 ziBJvOq794LRS94F7cXZWvv*=4l#cwFSdh;E*{Adwa)TAt zlWqCRPU#lVYE{K#Sfj<4-b1{j=1+_4QMo9WrhyeZE}Hg54~yJ5rXUWIJWNHqQB(uH zs`fV*2aQ+HOgyO9Vo&+AIPmFGm1;k!C$kGa%82I zXxNow96lD zAJ~>-_OBNb>YU!*xkqxj$67OHh)-kCZ?jI^+IoU8kNXyP7t#_z6om0{)m}1RAK#`U z{HlMab+~9vz-BuOibM%+@nwc;#Ufe}BYC~w+DQCfRwJo=4%2~1bRV#=G$!a&+B>8H zWa1}P49IDf0bx`if-Me9SkpAmVe|f+y5y~^mwNapI6k~&Mj^7vktJ`i^p2MztXOFW zy`VsV)L08(9o@OUvwiTVp(R^kY|wpcWvvC-)g=06zNsgz^I%(NHci}_suCkCy|V|1 ziLKC%9)di<kIf-QT1Y6}BM++wBT7{z+^K413q6{i%;=(7kJEC>)@_-F8}HIb_dla1bmE+oc`f+~erzET(yDrzKd^%80T3Gk|UA>(~=&)-#&K^W{e|Hc5>8 zYoO&LH(Ihi(BWT#Rs9h)D?BeTM#{NQv(8Hfsb)DSTV3~J*KsL0^a5@v$sc+lw&yrW znX|^vZUX$xKX}c#NCtm;WA0=BIcemN;QT7y6DGAJ&5g_y=lOJU5uJI2NlxkIQq8JL0K4s7TpSkAhGBcEK91H+&ceEP2~ZBn9hL!=7+?y$ z_jg{$xV=Bj?9}bj+89V8miNM_Wty%Gg;%D_EuE6zFR1q(%_qm44;s0pRs!dlKl0b; zV=PBxZMG4n7;4Tl;N+sI{NCaHB_MC1vIWWc-kq5;;!Gsky~s002Ok6WKVumGJdTJu zJmM2Z+w}uAK@ow2+hmYR7r>ejLCS6-I!E=FTxEy@SV%+|mw*asbg3rNg*`qskgn(O zPQjpz2>U0*qD>p-MFf@w^0&x_<^*4vNogjESj7fQ55$Qk&pp{*|C4jGGXFNl4W$zS z_b6u8fV#5ESxP56T+cnCC7nt_tw-{>vy2#KEYJ}jWi&YglS!d!8q?^hktG*u%CC^& z5Q8$XBc-^w5RkV62-RxdV1(qfZ{t5Z0Z}CBNuMM@vEPji5fc?NtGAH*Un(>F zwY4{qWX5vDA=o~UG27SIVRnW4`LZ<&;q4PH5oy<9lFcv!keL^o>^z;5Py79EH}c{` za4WTr5nX0HICzk=%qVI~a3oJCvejnGj6BKyyGw&1@&~&lQi1w%dBuDm-X+^##gRZQ z`Q8OTiNph+ZW~_@ohaRii?Z^%&w?uq7+OD(h7X0ApS+6CJ1SvZllv2JAPOj79(-00 zXjlc=v(8}cG(VDZl(aBE|NG0~xT;?uEQnQ$D&lUeX*;7APGeYb+_Y75|F0we_rQSz zMzQrdz!IoKmDitgFQ@ioRjfI$l=O-S4d==7>vEh6Z?gWt6WHiUn%`jujLL7#!v;#Gp%VtJ<}cl zya2oMjel_p>P>o?O>@@ufRGZ)GgdaItbqRnSdg8>LHG^2?*T5|g)NeR+feBWtH~OpBRHJ6(qae1Qk)wmS=N{_9^-*&iSK#lH=~??_U&g~vfT}js zrQG+Xc;w7(fk?*tL<>W!x@27M?SwmIc22abFO?^1B=Y&}_f^$tBCbrfQstx?#6X72 zoP?=2Ws2Dj1kq#N+~V~K*~`E9WF%bRzxI7@sYoN|uaKYO)1x7=D|!@J)$HGLcXFgs zXMT4MjAXJ{$XJJGQTgxmS}F_G4bJxpOi04?y4S#nW>6S<4NF16-s?~w3r0aj3r?n| za!cuZXUiik!CkKt1riNGW@#6ylGrUSdXNV48UuR@kc~0uyY&DXZ zaB>JFiYf4pNZwN$kmVdfOW+^mK*DLfiC92AvK<%&pg~{I=FiMmB9$xOrV%S_R3##q z#KE1Is1$0Dsv-}bg(@;FxMsV#g)EFoJCy$`k&lg^Gwa%2DwxG&V)$V+Q| zTo0$clD~Q@bm9@6dse80;^udFevKX72(xjhW_qcTZEtHc^1gcWtLO~MCBmykF{4(G zizz*PQ`WKZ7b->px3i~M$B1B~$$AHuU(k^ig0-uUed<9X@4(k0az*@{JQ=8r;Nz$V zwmD=&K^N0cub4%>Jd@KHCZ2H|p7Wg>b_Q-nMaDgF86iVaWj15e1GW4Dlh@B7{TfEK zij)9G={?hFLM2qU@4T%;_MHdtV!g>lzDnB$x_7s&S&@sKT%eV~(gZkiUnhQV-o%uN zn1Tx>+Y#NxXD-0t#Cm(QYyB@(Q(L~~olcNu$KB>8c8WM!Ze3rT6tq!*#NvXof0a=3 z8q_Eps7^T3E32GRF@A5U$$>A-<~@W<*t*h-HukG8dBlk_0t~vZV=z*!0EER;RMYGa zah3m>b9;T4vG|3cyHf}{E^a~t-MOcGwBo}s)W6N|cXZYKc~*E=KoF#(Ky3kY6fnh{ z07xQck7)}o`0kbQ2*<;8(|sg2m7DvJwCldj*Iv|R&AKygFrg*@cM5t(Iw^$!nz(1v z7>@S6aeO6E%1O46PB{x>>DFL?Cc8wbUDjfualK!NkIO|GDk(i+Bl$HnKNhq5tnty( zU`z8mx8G^@eT@m6e_~+p+$+ia_0qiH&epsj3num*9aUoWTl$3>y&T8$p4}>{wKNPO z;k-TLSq3`-RJUtaVtPFBW)l&L23m8XQT#%zE{ANJh*BJ9iVww9V~1SMuUgS-k3NHO zXaIqwjx%GDG#AiMe2_KxcY$0P`?FqsdjvwummH2H=4Y>{f1zMav;}_rE}>@`c|-qMHmaG7wI4X)FDZe*x1&5k;V+TS zx;8-?AnC$+qR?kZfjAFIDMttW3AW(UoYWdwwPzktjQ=pZCA*alDw*Di!_29b33qMZ z0xfjQ%XehM?M|dd$w0CCATi-7^Ci)W=3?kQb7L8EoZrHaAVtQ7L@JcUneB9QFm`Uk zZ;tnhEpV3RQzR%9Cy_nAg;0XJITm|bum?+> zc1IuRtO`4Gvc(LewQI<%0b!u`FUVSYr|`FmK&H`}h>UDzz|G&Sng|vVYV&AGllb-Z z$J(!Kt8Y%EcJMGeJmh5JUE4y!>Rt$pDAA|DS^_)fs6GS8R3Sx_3s-02K0nC(FJqE+T1#xGZoQ zH5H85jAr33Uav^FxbaoPXQ1jLHIWzlvw^bX`$gseH+RrxYhdtVNqTPh<$ym|v_UAW zHLt2;X(|k+C`)dk0zL0s%F-t(U7GKXZGa)FWGJ*|lK=_9Mb2l!N8(<=)N7e3X{{bqy!HZ({n2aB5I zqrpmrmQAzGJ8EI|FOC_sRGf(3GRe1HIc*ESRkzHrAb+u!EYs?|7p=x<`r1V9wXjJu zd&o8nd&mlMAaCyit;WVv^tKg8As?tK@wF&6WwTa+`1--~x@(F5O}A`AQFcqR?GT0d zw}KWSXfl$l7N5Hs!v|Aaed_c>5z8-|263e7iugiy># z7`dtKuD+77VBeJ)_ySXiq=JxJW(P&x_ncjpzRx7g_0SyM-3p zjqSb*X8IQoO=NifEyj?!Dy8-$`Q!zw3b|H6v5OSt zjEgh-ax!Q8Mt37)Tg`VJ3Hbm!C0kOkApezQl-@8g>SN7HFosH=WuU4u`&ZJ!VSv93 zD|}AolK<_=NGey3excl)uwXJq^Wef+5_cYF2;=sYsml;qu`rQZlo?e~KFj4{SIoBI zPkDJK@DI9Z6W)N1*p|_L%2CVlg@GB3#i7`r1f(kIN}|d{(1bH}t~tM$4v|oBw^mDM z(Jg?3l4w(!EgX^Ixnr?RWuJ|_ICsYd%B&UCV9ChoHmE{|{=$Wq$QUj~%D&n8kNqc% zq|k>w^`u$U8mD)J)-D}J2m$t)6yR0cth*jTz%JS_B`pX%zT zB#ilDO|dnv$ysncQ7K3gj5h?R#`p_Nu%#ncp3~%3Bis}xQfcxOX9YU*O(eQ!wHy(j)Pzn>$;#q_q%EQZmt7uL6m)@sT?DQD+oaqh zU_o=djHI5PtOJI1wL$?kwz~o*o|ggK@eDQ#e^dgH;pQZc7746<{3?Fhu;(`cgh)RX zsBivL2xb2~MU;JOdDE`piK?xSw6ZxOhrZc(gC?uekr#rg&Ya^ZgN3Xu>CJ5GCK zi;`l`ggS^>vC(c9p$^&?suO+Ulw{iqU+qC2!adOf{70R(Oi&!Fzof#X5Pm$U3vI6o z3#O*sAbtEdO8-aDN=E4k6H;p3X z2ZOCV$aAK;5HL?gsL24ls;n&5`?x%Rn3HCQz4!u4O+UttnAersbh|zu(mLrEZ{WQ{ z>$23k;2#Jgc?@0V?%kMigWf-5dX{3!Y&dYuT(UK%rXn3y=IJcAP z6ULgVqEF+T8$|33V^wA>pcpA}?>S}OQ4r6$4DaMse4o%)ot!|wWdpb$)r>{Mf2{snvyr+_hmkYtr@yI-P`oV{ts^}(Q1haBu)i#fd{>pP?4D}Nb9tvg9ID0uuVBj@(1H1@{ z`v+DU7B65Zo;KYZXO2j-b%PeArXl!4a0O62Pjz85$)$aRs_WKzO}IXc^H*%edB=^= zk4t$@NAuL81du z_?WimUYJZYPGW8y3~)Fv874(?T0}gmb^uTZ_t%da717|}{nvhu40t2JZvtZ6S|=7o z$yWuq0qH@_^C{_~GjjR;6fQ_+dgY?tW{0|CDEl=aShrL91(y-q3$;DU=Ue>;tWe{& z`!(x00{nGJ0A69+;o^<}=uDD0@|P-fuM1aPK=x+#Qt4c1*$xs?C2jk`aXW=o?Y#zx3qFE3nT}38Qo=F1d{H^9W-&u@_8lAy&T97s@a4Q{=@pF z_`$WtJX`xnRNAC}Iy-NStPB5v$40aaVq#Sr+(uIl0<=8Y4o85xY@uBPoJ29%jrq>R zcud3Ub%gwjs6xI_>(gGf`QWhVDiF;1)j9eZRHkG7MipKy5|Y*@D#^HndNJbJZU7!% zPPugW7DN+dnxSzvq0~UTJk8~Vk=f~9VnVnHZ#J3}-sn^q-0KUu{i9KezKk$pm_Q}U z+W832*GMFBUweTT-i~E^;y4Pru>a#IY1FHDE#)@uset*hAfYI1}d0?4S}i-owJ+IzO+?Etv^vm+yV_#YNmSYv*k|*|1lbQvmzSD z3dFxQ4t9#K4GZahDj;3;%=w^+*MP&zAE4P@-hBPc zKH(Z(|LBF0;<^W&hzP6d5W<}Kd$+y71+8De4++u?A*hRO>wHIQ+6Is;cMH1c;j9vW zrem7vs^B+qA>LoJ$y|RVF#Qc4gd5@LtY1zd+VbkKH=N#{0L!fhn90G;f6~X|O@W>? zMfxcPb}JbJ22N^3BY^S7@h^mWSDta>&-cY@Wi`hAfPp(tbUg(S8R)5})qEtp8oKO} zW;1lel93RmPij@hzpN|IS(Rg`Tite?$+Mz~yq<9UPU$^Wf8irw%Id9DRcW0j{Po7= zf)^2F9{oTcr2V62uIw!3d9Vldd^-G%PiN-S8H|iefdH}uvlFsk#BKhSSx9Ax&){@R zEwz*4c7^)%%?FEmo!ERvgA=K_B_+VuJ8Sx%4`}EdqBU&CnRH+$D-9=zk|}~y)&)>A zs#MX16KuJ(^GK5W`K}3{vZY6u-g&eVW*L3|$qacBh4R;?-&5%fZkLTxsVcthX$MK5 zoXAktzKvPh`A4(2{~X2}AZC3fLU@C@OcV|}g99LpU*xw^vF`?LV{{o9Ea8$GQV5>( zQpf7i)A^YSaSC*->SPrSn3`$lUo`F9uGuhBbXVZ3d!+Lu;%Z8%OHv=OvNTtaV1#e_F>w6Dj=A&> z2TQlc?eeVzYko0p|6dG$=N3Z>de`HDrYaZs(Mf%J2_aN(lp-^Th=!J%%(?OAyzPx- z09pGU29B{us!%_MuaOWBsxa`B0?Mu2?~sy=ifYAJfnhAD=VPmu$n(XpIVfrq@ny5y z$v8KP<_zZCI&pKBOJ7w=u`jt2@H_%k9@uZlM+uS!3+^m<9$b(AgY^$H@c$mq^h}$bZFb;vfEo* z?BuD$7BAjZ;_3H566oh@ect|D6$<_hED~N&7vVHvBv3|#^PIZQ^Ydx(5Am=Pvl3S8 z^@8@(DS(b;d_;h8n9A^dJPh9EI$-#}Q^_8rc7HUV7sWtsB ztDmJ2ut7tUkj;L1dQpT7?3Q1s<^Y2&UBM?oJEBWd6U}=sll{buCxDe0-pvF{oygOSs_m-B2_Lq`ow|u0}4vT-7@cz$<^8Mcr z7GvzrtZ&;$3=lQS6uIMN$m!Mly${un$l_=)tCYUFuDmd>aVTQ) zC&wCL0(c$UgEnNOzXUzd_nmy$_{jskC7ECL@SfR(rRLFpLWGY;8WM!H-GwE=Ul{?M zEo*%ZV+yQg`22pmMkWWT1+&?Y-54tzOuPU;lMLdA0_eFIJ&uvU5$h4CMogj>b!w~d z6_*;0GpI9RQbzbV*PZDNTdlgN_QO@3#8|Hefj^CP;E^uOsF}3(EA-%Rb_jm}iXMza z_C9?PlF-(P?UR}M8moeNICAy;L6qyrIhvZ(V<&Xgp$FZ)2Cdve^Zby3JI;emo!p6* z=gL7CZp4E{LAQ8fHX{Nz==+xiZbbMNAlBXs^P!vzT{SU0SyJ~c=B%D4pD4u3V&MBZ zYo_u_)4+_jW()ec4GFFUD5nkA)LTILK=yMvSkwy!Yb4Cs*11FCqc!^kkKO}PbO9Cg ze+R3h{wl@s*yaeIq2yX%sYA6A2V#wWcY7S7+Z`?!!Z<45Do_)y!&z@oWx6_91Ygmh z601HwH_rSPUrK;Wgqi^IMv`%^y@>sweL?|fNI=^-?KT4s{It(U$W&YrRL=wTJHVLA}Ema|^@P`H%4!*D?^g z!x*eLs1G(+$d0nChAgHS{0v-3DN~I#>$1QIo z`U_3OD5B@CYiivo$BnD_T;bUj8l1jE*C}FXN*kC1?pLCRz6qB;@7k6CsmF2N#H{uf z9icr)v2}Bo+MV>UiOyDM&{fz7RGFK^Nt|mfCIaX4YZ>)~F1zvI!32Yg6;5P-(htwc zrnEx3JBpIuqDPyhP~-MDoB!1FHww$$m6X=CPa|ziL7tUZcp?)|*DX`hyPD#cva|VM z1&=revQd%lMevvF(&f)Anp+JC!-D3=JyH&<8(?N%&?WPf`2fo2JchAl{hvMEqcY8q z>o6)X=qiB$ENlCz;b=h^w}~SLt|oTJS+IhSH(n-4A$37?WnaW!e_0aZcA25J?jPt5 zJ9bCaW@wBR&!soHT03$uUB?ogou80~PSl0=>k=aiX+Nwugr^nj&!z=d}_aXF5Enz4zOZlC^gsbk))zN!z zHn1AubZL-uLyV%^@oVy?DeaC|JOBWAnHtm1#%s0!0o%s}fUV*L0000$=A^Y07dtQm K000001X)^w5Jv?7 literal 0 HcmV?d00001 diff --git a/benchmarks/ComputerSystems/IndexOptimization/README.md b/benchmarks/ComputerSystems/IndexOptimization/README.md deleted file mode 100644 index 614c0d39..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/README.md +++ /dev/null @@ -1,64 +0,0 @@ -# IndexOptimization — PostgreSQL Index Selection - -Optimize index configuration for a TPC-H SF1 analytical workload on PostgreSQL 16. - -## Benchmark ID - -`ComputerSystems/IndexOptimization` - -## Task - -Given a PostgreSQL database with TPC-H schema and a mixed analytical SQL workload, find a high-quality B-tree index configuration that improves query performance compared to a heuristic baseline, subject to index count and storage constraints. - -## Economic Relevance - -Database index optimization is a core database administration task with direct operational impact: -- PostgreSQL is one of the most widely deployed open-source databases in production, used by enterprises across finance, e-commerce, logistics, and analytics. -- Unoptimized indexes cause unnecessary storage costs, slow down write operations, and degrade query performance — a poorly indexed database can be 10-100x slower than a well-tuned one. -- Automated index selection reduces the need for manual DBA tuning, which is both expensive and error-prone at scale. -- This benchmark uses TPC-H, the industry-standard decision-support workload, making results transferable to real-world analytical processing environments. - -## Workload - -6 TPC-H SF1 queries covering multi-table JOINs, aggregation, range filters, sorting, and sequential scans. - -## Environment - -Requires Docker: - -```bash -docker build -t frontier-pg-index:latest verification/docker/ -``` - -## Quick Run - -```bash -python scripts/init.py --input data/raw_task.json --output outputs/candidate.json -python verification/evaluator.py scripts/init.py -``` - -## First-Time Setup - -Generate TPC-H SF1 data (required once before running the evaluator): - -```bash -bash data/tpch_sf1/gen_data.sh -``` - -## Unified Task - -```bash -python -m frontier_eval \ - task=unified \ - task.benchmark=ComputerSystems/IndexOptimization \ - task.runtime.isolation_mode=docker \ - task.runtime.docker_image=frontier-pg-index:latest \ - algorithm=openevolve \ - algorithm.iterations=0 -``` - -## Algorithm Agnostic - -This benchmark is algorithm-agnostic. Rule-based, ML-based, LLM-based, and search-based approaches are all welcome. - -Inspired by research on automated index recommendation including LLMIA (Zhao et al., arXiv:2503.07884, 2025). diff --git a/benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md b/benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md deleted file mode 100644 index e64aaaba..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/README_zh-CN.md +++ /dev/null @@ -1,38 +0,0 @@ -# IndexOptimization — PostgreSQL 索引优化 - -对 TPC-H SF1 分析型 workload，在 PostgreSQL 16 上优化索引配置。 - -## 统一 benchmark ID - -`ComputerSystems/IndexOptimization` - -## 任务描述 - -给定一个 PostgreSQL 数据库（TPC-H schema）和一个分析型 SQL workload，在索引数量和存储空间约束下，找到比 heuristic 基线更好的 B-tree 索引配置。 - -## 环境准备 - -```bash -docker build -t frontier-pg-index:latest verification/docker/ -``` - -## 快速运行 - -```bash -python scripts/init.py --input data/raw_task.json --output outputs/candidate.json -python verification/evaluator.py scripts/init.py -``` - -## Unified task 运行 - -```bash -python -m frontier_eval \ - task=unified \ - task.benchmark=ComputerSystems/IndexOptimization \ - task.runtime.isolation_mode=docker \ - task.runtime.docker_image=frontier-pg-index:latest \ - algorithm=openevolve \ - algorithm.iterations=0 -``` - -本 benchmark 对算法无关，基于规则的、ML-based、LLM-based、搜索算法均可参与。 diff --git a/benchmarks/ComputerSystems/IndexOptimization/Task.md b/benchmarks/ComputerSystems/IndexOptimization/Task.md deleted file mode 100644 index a44d3937..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/Task.md +++ /dev/null @@ -1,83 +0,0 @@ -# IndexOptimization - -## Overview - -Find high-quality B-tree index configurations for a PostgreSQL database serving a mixed analytical SQL workload (TPC-H SF1). The goal is to improve query execution time under index count and storage constraints, compared to a heuristic baseline. - -## Input - -The candidate script (`scripts/init.py`) reads the task configuration from `--input` and writes output to `--output`. The input provides: - -- **Workload queries**: SQL queries with structured metadata (tables, filters, JOIN conditions) -- **Database schema**: Table definitions with column types, row counts, and existing indexes (primary keys) -- **Constraints**: `max_indexes` (10), `max_storage_mb` (500) - -## Output - -```json -{ - "indexes": [ - {"table": "orders", "columns": ["o_orderdate", "o_custkey"], "method": "btree"} - ] -} -``` - -## Scoring - -| Metric | Description | -|--------|-------------| -| `no_index_time_s` | Total query time without indexes (reference) | -| `baseline_time_s` | Total query time with heuristic indexes | -| `candidate_time_s` | Total query time with candidate indexes | -| `speedup` | baseline_time / candidate_time | -| `combined_score` | log2(speedup) × (1 - storage_penalty - count_penalty) | - -**Hard constraints**: All queries must return identical results; ≤10 indexes; ≤500 MB storage. - -## Workload Queries - -| ID | Pattern | Tables | Index optimization target | -|----|---------|--------|--------------------------| -| Q1 | Single-table aggregate | lineitem | Sequential scan test | -| Q3 | Join + filter + aggregate + order | customer, orders, lineitem | Multi-join, range filter | -| Q5 | Multi-way join + aggregate | 6 tables | Star schema join | -| Q6 | Single-table range filter + aggregate | lineitem | Selective filter | -| Q10 | Join + aggregate + order | customer, orders, lineitem, nation | Join + filter | -| Q12 | Join + dual-condition filter + aggregate | orders, lineitem | Filter selectivity | - -## Constraints - -1. Only modify `scripts/init.py` — the ONLY editable file. -2. Keep CLI contract: `--input` and `--output`. -3. Output must contain an `indexes` list. -4. Each index must specify table, columns (list), and method ("btree"). -5. Maximum 10 indexes per submission. -6. Total index storage must not exceed 500 MB. -7. All queries must return identical results before and after indexing. -8. Do not recommend indexes already covered by primary keys (listed in schema existing_indexes). - -## Economic Relevance - -Database index optimization directly affects production system performance and operational cost: -- **Query performance**: A well-chosen index can reduce query time from minutes to milliseconds in analytical workloads. -- **Storage cost**: Each unnecessary index consumes disk space and memory for caching. At TPC-H SF1 scale, an unused index costs ~10-100 MB. -- **Write overhead**: Indexes slow down INSERT/UPDATE/DELETE operations. In read-write mixed workloads, excess indexes degrade overall throughput. -- **Operational complexity**: Production databases often have hundreds of tables. Manual index tuning does not scale — automated index selection is a recognized industry need. - -This benchmark evaluates an agent's ability to make these engineering trade-offs: improving read performance while respecting storage and count constraints. - -## Optimization Directions - -Agents can explore the following strategies to find better index configurations: - -1. **Identify high-selectivity filters**: Indexing columns used in WHERE clauses with high selectivity (e.g., `o_orderdate`, `l_shipdate`) can significantly reduce scan ranges. - -2. **Covering indexes for JOIN columns**: Indexes on foreign key columns (e.g., `o_custkey`, `l_orderkey`) can accelerate hash join probe phases. - -3. **Avoid over-indexing**: Unused indexes incur storage and maintenance costs. The scoring formula penalizes both storage ratio and index count — more indexes do not always yield better scores. - -4. **Recognize sequential-scan scenarios**: Queries like Q1 (aggregating most of a large table) may not benefit from indexes. Over-indexing such queries adds cost without benefit. - -5. **Multi-column indexes**: A composite index (e.g., `(o_orderdate, o_custkey)`) can serve multiple query clauses simultaneously, potentially replacing several single-column indexes. - -6. **Analyze Q1's behavior**: Q1 performs a full table scan on `lineitem`. Indexes do not help this query but consume storage. The scoring formula accounts for storage cost — a good strategy should recognize when not to index. diff --git a/benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md b/benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md deleted file mode 100644 index 367449f1..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/Task_zh-CN.md +++ /dev/null @@ -1,53 +0,0 @@ -# IndexOptimization - 数据库索引优化 - -## 概述 - -在索引数量和存储空间约束下，为 PostgreSQL 数据库上的分析型 SQL workload 找到高质量的 B-tree 索引配置，以优化查询执行时间。 - -## 输入 - -candidate 脚本通过 `--input` 读取任务配置，将输出写入 `--output`。输入包含： - -- **查询 workload**：SQL 查询及结构化 metadata（涉及表、过滤条件、JOIN 条件） -- **数据库 schema**：表定义、列类型、行数、已有索引（主键） -- **约束**：`max_indexes` (10)、`max_storage_mb` (500) - -## 输出 - -```json -{ - "indexes": [ - {"table": "orders", "columns": ["o_orderdate", "o_custkey"], "method": "btree"} - ] -} -``` - -## 评分 - -| 指标 | 说明 | -|------|------| -| `baseline_time_s` | heuristic 索引下的总查询时间 | -| `candidate_time_s` | candidate 索引下的总查询时间 | -| `combined_score` | log2(speedup) × (1 - 存储惩罚 - 数量惩罚) | - -## 约束 - -1. 只修改 `scripts/init.py` -2. 保持 CLI 接口不变（`--input` 和 `--output`） -3. 输出必须包含 `indexes` 列表 -4. 每个索引需指定 table、columns 和 method -5. 最多 10 个索引 -6. 索引空间不超过 500 MB -7. 索引前后所有查询结果必须一致 -8. 不要推荐主键中已存在的索引 - -## 优化方向建议 - -Agent 可以尝试以下策略来找到更好的索引配置： - -1. **高选择性过滤列**：对 WHERE 子句中选择性高的列（如 `o_orderdate`、`l_shipdate`）建索引，可显著缩小扫描范围 -2. **JOIN 列覆盖索引**：对外键列（如 `o_custkey`、`l_orderkey`）建索引，可加速哈希连接的探测阶段 -3. **避免过度索引**：未被使用的索引会产生存储和维护开销。评分公式中存储惩罚和数量惩罚共同作用——更多的索引不一定带来更高的分数 -4. **识别全表扫描场景**：Q1 对大表大部分行做聚合查询，索引无法帮助此类查询。过度建索引只会增加成本 -5. **多列索引**：组合索引（如 `(o_orderdate, o_custkey)`）可同时服务于多个查询子句，可能替代多个单列索引 -6. **分析 Q1 的特性**：Q1 对 `lineitem` 做全表扫描，索引对此查询无帮助但消耗存储。评分公式考虑了存储成本，好的策略应能识别何时不应建索引 diff --git a/benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py b/benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py deleted file mode 100644 index 2f19c605..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/baseline/heuristic.py +++ /dev/null @@ -1,130 +0,0 @@ -""" -Heuristic index recommendation baseline. - -Strategy: -1. Collect all WHERE filter columns and JOIN columns from workload queries. -2. Remove columns already covered by existing indexes (primary keys). -3. Count column frequency across queries. -4. Create single-column B-tree indexes on the most frequent columns. -5. Stop before exceeding max_indexes. -""" - -from __future__ import annotations - -import json -from collections import Counter -from pathlib import Path -from typing import Any - - -def load_json(path: str | Path) -> dict[str, Any]: - with Path(path).open() as f: - return json.load(f) - - -def _resolve_queries(task_dir: Path, raw: dict[str, Any]) -> list[dict[str, Any]]: - queries: list[dict[str, Any]] = [] - for ref in raw.get("queries", []): - qpath = (task_dir / ref).resolve() - queries.append(load_json(qpath)) - return queries - - -def _flatten_filter_columns(metadata: dict) -> list[tuple[str, str]]: - """Extract (table, column) pairs from filter conditions.""" - result: list[tuple[str, str]] = [] - for f in metadata.get("filters", []): - tbl = f.get("table", "") - col = f.get("column", "") - if tbl and col: - result.append((tbl, col)) - return result - - -def _flatten_join_columns(metadata: dict) -> list[tuple[str, str]]: - """Extract (table, column) pairs from JOIN conditions (both sides).""" - result: list[tuple[str, str]] = [] - for jc in metadata.get("join_conditions", []): - for side in [jc.get("left", []), jc.get("right", [])]: - if len(side) >= 2: - result.append((side[0], side[1])) - return result - - -def _columns_in_existing_index(table: str, schema: dict[str, Any]) -> set[str]: - """Return set of column names already covered by existing indexes on this table.""" - covered: set[str] = set() - table_info = schema.get(table, {}) - for idx in table_info.get("existing_indexes", []): - for col in idx.get("columns", []): - covered.add(col) - return covered - - -def recommend_indexes( - queries: list[dict[str, Any]], - schema: dict[str, Any], - constraints: dict[str, Any], -) -> list[dict[str, Any]]: - max_indexes = constraints.get("max_indexes", 10) - - # Collect candidate columns from filters and joins - candidates: list[tuple[str, str]] = [] - for q in queries: - meta = q.get("metadata", {}) - candidates.extend(_flatten_filter_columns(meta)) - candidates.extend(_flatten_join_columns(meta)) - - # Count frequency and deduplicate - freq = Counter(candidates) - - # Sort by frequency (descending), then by table name for determinism - ranked = sorted(freq.items(), key=lambda x: (-x[1], x[0][0], x[0][1])) - - # Build index list, skipping columns already covered by existing indexes - result: list[dict[str, Any]] = [] - seen_pairs: set[tuple[str, str]] = set() - - for (table, column), _count in ranked: - if len(result) >= max_indexes: - break - if (table, column) in seen_pairs: - continue - # Skip if column already has an index - existing = _columns_in_existing_index(table, schema) - if column in existing: - continue - seen_pairs.add((table, column)) - result.append({"table": table, "columns": [column], "method": "btree"}) - - return result - - -def main() -> None: - import argparse - parser = argparse.ArgumentParser() - parser.add_argument("--input", required=True) - parser.add_argument("--output", required=True) - args = parser.parse_args() - - input_path = Path(args.input).resolve() - task_dir = input_path.parent.parent - raw = load_json(input_path) - queries = _resolve_queries(task_dir, raw) - schema = raw.get("schema", {}) - constraints = raw.get("constraints", {}) - - indexes = recommend_indexes(queries, schema, constraints) - - output_path = Path(args.output) - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text( - json.dumps({"indexes": indexes}, indent=2), encoding="utf-8" - ) - print(f"heuristic indexes: {len(indexes)}") - for idx in indexes: - print(f" {idx['table']}({', '.join(idx['columns'])})") - - -if __name__ == "__main__": - main() diff --git a/benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt b/benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt deleted file mode 100644 index e16311a1..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/baseline/result_log.txt +++ /dev/null @@ -1,36 +0,0 @@ -IndexOptimization baseline (heuristic) result log -================================================ -Date: 2026-07-11 -Evaluator: verification/evaluator.py (evaluate() function) -Candidate: scripts/init.py (baseline: returns empty index list) -Environment: Docker (frontier-pg-index:latest), TPC-H SF1 - -Results: --------- -valid: 1.0 -combined_score: 0.632 -no_index_time_s: 7.0 (no indexes, reference only) -baseline_time_s: 9.8 (heuristic: 10 single-column indexes) -candidate_time_s: 5.9 (candidate: 0 additional indexes) -n_indexes: 0 (beyond existing primary keys) -storage_mb: 237.6 (primary key indexes only) -data_size_mb: 1312.9 (TPC-H SF1 ~1.3 GB) -correct: 1.0 (all query results match) -speedup: 1.67 (candidate vs heuristic) - -Observations: -------------- -1. The heuristic baseline (10 indexes) is SLOWER than no-index (9.8s vs 7.0s). - This demonstrates that naive single-column indexing on filter/join columns - can degrade performance through unnecessary overhead. -2. The candidate (0 extra indexes) outperforms heuristic by 1.67x. -3. A good agent should find a configuration that: - - Selectively indexes high-impact columns (e.g., l_shipdate, o_orderdate) - - Avoids indexing columns where sequential scans are preferable (Q1) - - Stays within the 10-index and 500 MB constraints - -Heuristic index list (10 single-column): - orders(o_orderdate), lineitem(l_shipdate), orders(o_custkey), - customer(c_nationkey), lineitem(l_receiptdate), supplier(s_nationkey), - customer(c_mktsegment), lineitem(l_commitdate), lineitem(l_discount), - lineitem(l_quantity) diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json b/benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json deleted file mode 100644 index f0e0e701..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/raw_task.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "benchmark_id": "index_optimization", - "description": "PostgreSQL index optimization on TPC-H SF1 workload", - "queries": [ - "data/tpch_sf1/queries/tpch_q1.json", - "data/tpch_sf1/queries/tpch_q3.json", - "data/tpch_sf1/queries/tpch_q5.json", - "data/tpch_sf1/queries/tpch_q6.json", - "data/tpch_sf1/queries/tpch_q10.json", - "data/tpch_sf1/queries/tpch_q12.json" - ], - "schema": { - "nation": { - "columns": [{"name": "n_nationkey", "type": "integer"}], - "row_count": 25, - "existing_indexes": [{"columns": ["n_nationkey"], "type": "primary"}] - }, - "region": { - "columns": [{"name": "r_regionkey", "type": "integer"}], - "row_count": 5, - "existing_indexes": [{"columns": ["r_regionkey"], "type": "primary"}] - }, - "part": { - "columns": [{"name": "p_partkey", "type": "integer"}], - "row_count": 200000, - "existing_indexes": [{"columns": ["p_partkey"], "type": "primary"}] - }, - "supplier": { - "columns": [{"name": "s_suppkey", "type": "integer"}], - "row_count": 10000, - "existing_indexes": [{"columns": ["s_suppkey"], "type": "primary"}] - }, - "partsupp": { - "columns": [{"name": "ps_partkey", "type": "integer"}, {"name": "ps_suppkey", "type": "integer"}], - "row_count": 800000, - "existing_indexes": [{"columns": ["ps_partkey", "ps_suppkey"], "type": "primary"}] - }, - "customer": { - "columns": [{"name": "c_custkey", "type": "integer"}], - "row_count": 150000, - "existing_indexes": [{"columns": ["c_custkey"], "type": "primary"}] - }, - "orders": { - "columns": [{"name": "o_orderkey", "type": "bigint"}], - "row_count": 1500000, - "existing_indexes": [{"columns": ["o_orderkey"], "type": "primary"}] - }, - "lineitem": { - "columns": [{"name": "l_orderkey", "type": "bigint"}, {"name": "l_linenumber", "type": "integer"}], - "row_count": 6001215, - "existing_indexes": [{"columns": ["l_orderkey", "l_linenumber"], "type": "primary"}] - } - }, - "constraints": { - "max_indexes": 10, - "max_storage_mb": 500, - "statement_timeout_s": 60, - "query_repeats": 3, - "warmup_rounds": 2 - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh deleted file mode 100644 index 8b415fad..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/gen_data.sh +++ /dev/null @@ -1,114 +0,0 @@ -#!/bin/bash -# Generate TPC-H SF1 data for IndexOptimization benchmark. -# -# Usage: -# bash data/tpch_sf1/gen_data.sh -# -# This script: -# 1. Checks that Docker and the frontier-pg-index image are available -# 2. Clones dbgen and generates TPC-H SF1 data files -# 3. Starts a PostgreSQL container and loads the data -# 4. Creates a pg_dump file for fast evaluator restore -# 5. Cleans up - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -DUMP_FILE="${SCRIPT_DIR}/tpch_sf1.dump" -SCHEMA_FILE="${SCRIPT_DIR}/schema.sql" -DBGEN_DIR="/tmp/tpch_dbgen_index_opt" - -# Prerequisites -if ! command -v docker &>/dev/null; then - echo "Error: docker is required but not found." - exit 1 -fi - -if ! docker image inspect frontier-pg-index:latest &>/dev/null; then - echo "Building frontier-pg-index:latest image..." - REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" - docker build -t frontier-pg-index:latest "${REPO_ROOT}/verification/docker/" -fi - -# Clone or update dbgen -if [ -d "${DBGEN_DIR}" ]; then - echo "dbgen already exists at ${DBGEN_DIR}" -else - echo "Downloading TPC-H dbgen..." - if command -v git &>/dev/null; then - git clone --depth 1 https://github.com/electrum/tpch-dbgen.git "${DBGEN_DIR}" - else - mkdir -p "${DBGEN_DIR}" - cd "${DBGEN_DIR}" && curl -sL "https://github.com/electrum/tpch-dbgen/archive/refs/heads/master.tar.gz" | tar -xz --strip=1 - fi -fi - -# Compile dbgen -echo "Compiling dbgen..." -cd "${DBGEN_DIR}" -if [ ! -f dbgen ]; then - make -j4 2>&1 || { echo "dbgen compilation failed"; exit 1; } -fi - -# Generate data -echo "Generating TPC-H SF1 data files..." -./dbgen -s 1 -f -echo "Data files: $(ls *.tbl | wc -l) tables" - -# Start PostgreSQL container -echo "Starting PostgreSQL..." -CONTAINER_ID=$(docker run -d --rm \ - -e POSTGRES_PASSWORD=postgres \ - -v "${DBGEN_DIR}:/tpch_data" \ - -P frontier-pg-index:latest) - -# Wait for PostgreSQL to be ready -for i in $(seq 1 30); do - if docker exec "${CONTAINER_ID}" pg_isready -q 2>/dev/null; then - echo "PostgreSQL ready after ${i}s" - break - fi - sleep 1 - if [ "$i" -eq 30 ]; then - echo "Error: PostgreSQL did not start within 30s" - docker stop "${CONTAINER_ID}" >/dev/null - exit 1 - fi -done - -# Remove trailing pipe from .tbl files (TPC-H format compatibility) -for f in "${DBGEN_DIR}"/*.tbl; do - sed -i.bak 's/|$//' "$f" - rm -f "${f}.bak" -done - -# Create schema -echo "Creating database schema..." -docker exec -i "${CONTAINER_ID}" psql -U postgres < "${SCHEMA_FILE}" - -# Load data -echo "Loading TPC-H SF1 data (this may take a few minutes)..." -for tbl in nation region part supplier partsupp customer orders lineitem; do - echo -n " ${tbl}... " - docker exec -i "${CONTAINER_ID}" psql -U postgres -c \ - "\\copy ${tbl} FROM '/tpch_data/${tbl}.tbl' WITH DELIMITER '|' NULL ''" \ - 2>&1 | grep -c "^COPY" | xargs echo -n - echo " rows loaded" -done - -# Create dump -echo "Creating pg_dump archive..." -docker exec "${CONTAINER_ID}" pg_dump -U postgres -Fc -f /tmp/tpch_sf1.dump -docker cp "${CONTAINER_ID}:/tmp/tpch_sf1.dump" "${DUMP_FILE}" - -# Stop container -docker stop "${CONTAINER_ID}" >/dev/null - -echo "" -echo "==============================" -echo "TPC-H SF1 data generated successfully!" -echo "Dump file: ${DUMP_FILE}" -ls -lh "${DUMP_FILE}" -echo "" -echo "The evaluator will automatically use this dump on restore." -echo "==============================" diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json deleted file mode 100644 index 9c56466b..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q1.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "id": "tpch_q1", - "sql": "SELECT l_returnflag, l_linestatus, SUM(l_quantity) AS sum_qty, SUM(l_extendedprice) AS sum_base_price, SUM(l_extendedprice * (1 - l_discount)) AS sum_disc_price, SUM(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, AVG(l_quantity) AS avg_qty, AVG(l_extendedprice) AS avg_price, AVG(l_discount) AS avg_disc, COUNT(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-09-02' GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus", - "metadata": { - "tables": ["lineitem"], - "filters": [{"table": "lineitem", "column": "l_shipdate", "operator": "<="}], - "aggregate": true, - "sort_columns": ["l_returnflag", "l_linestatus"] - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json deleted file mode 100644 index 7b2601fb..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q10.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "id": "tpch_q10", - "sql": "SELECT c_custkey, c_name, SUM(l_extendedprice * (1 - l_discount)) AS revenue, c_acctbal, n_name, c_address, c_phone, c_comment FROM customer, orders, lineitem, nation WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate >= DATE '1993-10-01' AND o_orderdate < DATE '1994-01-01' AND l_returnflag = 'R' AND c_nationkey = n_nationkey GROUP BY c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment ORDER BY revenue DESC", - "metadata": { - "tables": ["customer", "orders", "lineitem", "nation"], - "join_conditions": [ - {"left": ["customer", "c_custkey"], "right": ["orders", "o_custkey"]}, - {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]}, - {"left": ["customer", "c_nationkey"], "right": ["nation", "n_nationkey"]} - ], - "filters": [ - {"table": "orders", "column": "o_orderdate", "operator": ">="}, - {"table": "orders", "column": "o_orderdate", "operator": "<"}, - {"table": "lineitem", "column": "l_returnflag", "operator": "="} - ], - "aggregate": true, - "sort_columns": ["revenue"] - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json deleted file mode 100644 index a304db52..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q12.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "id": "tpch_q12", - "sql": "SELECT l_shipmode, SUM(CASE WHEN o_orderpriority = '1-URGENT' OR o_orderpriority = '2-HIGH' THEN 1 ELSE 0 END) AS high_line_count, SUM(CASE WHEN o_orderpriority <> '1-URGENT' AND o_orderpriority <> '2-HIGH' THEN 1 ELSE 0 END) AS low_line_count FROM orders, lineitem WHERE o_orderkey = l_orderkey AND l_shipmode IN ('MAIL', 'SHIP') AND l_commitdate < l_receiptdate AND l_shipdate < l_commitdate AND l_receiptdate >= DATE '1994-01-01' AND l_receiptdate < DATE '1995-01-01' GROUP BY l_shipmode ORDER BY l_shipmode", - "metadata": { - "tables": ["orders", "lineitem"], - "join_conditions": [ - {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]} - ], - "filters": [ - {"table": "lineitem", "column": "l_shipmode", "operator": "in"}, - {"table": "lineitem", "column": "l_commitdate", "operator": "<"}, - {"table": "lineitem", "column": "l_shipdate", "operator": "<"}, - {"table": "lineitem", "column": "l_receiptdate", "operator": ">="}, - {"table": "lineitem", "column": "l_receiptdate", "operator": "<"} - ], - "aggregate": true, - "sort_columns": ["l_shipmode"] - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json deleted file mode 100644 index 9e059ed1..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q3.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "id": "tpch_q3", - "sql": "SELECT l_orderkey, SUM(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer, orders, lineitem WHERE c_mktsegment = 'BUILDING' AND c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate < DATE '1995-03-15' GROUP BY l_orderkey, o_orderdate, o_shippriority ORDER BY revenue DESC, o_orderdate", - "metadata": { - "tables": ["customer", "orders", "lineitem"], - "join_conditions": [ - {"left": ["customer", "c_custkey"], "right": ["orders", "o_custkey"]}, - {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]} - ], - "filters": [ - {"table": "customer", "column": "c_mktsegment", "operator": "="}, - {"table": "orders", "column": "o_orderdate", "operator": "<"} - ], - "aggregate": true, - "sort_columns": ["revenue", "o_orderdate"] - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json deleted file mode 100644 index fb452b53..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q5.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "id": "tpch_q5", - "sql": "SELECT n_name, SUM(l_extendedprice * (1 - l_discount)) AS revenue FROM customer, orders, lineitem, supplier, nation, region WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND l_suppkey = s_suppkey AND c_nationkey = s_nationkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'ASIA' AND o_orderdate >= DATE '1994-01-01' AND o_orderdate < DATE '1995-01-01' GROUP BY n_name ORDER BY revenue DESC", - "metadata": { - "tables": ["customer", "orders", "lineitem", "supplier", "nation", "region"], - "join_conditions": [ - {"left": ["customer", "c_custkey"], "right": ["orders", "o_custkey"]}, - {"left": ["orders", "o_orderkey"], "right": ["lineitem", "l_orderkey"]}, - {"left": ["lineitem", "l_suppkey"], "right": ["supplier", "s_suppkey"]}, - {"left": ["customer", "c_nationkey"], "right": ["supplier", "s_nationkey"]}, - {"left": ["supplier", "s_nationkey"], "right": ["nation", "n_nationkey"]}, - {"left": ["nation", "n_regionkey"], "right": ["region", "r_regionkey"]} - ], - "filters": [ - {"table": "region", "column": "r_name", "operator": "="}, - {"table": "orders", "column": "o_orderdate", "operator": ">="}, - {"table": "orders", "column": "o_orderdate", "operator": "<"} - ], - "aggregate": true, - "sort_columns": ["revenue"] - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json deleted file mode 100644 index 118d52df..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/queries/tpch_q6.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "id": "tpch_q6", - "sql": "SELECT SUM(l_extendedprice * l_discount) AS revenue FROM lineitem WHERE l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1995-01-01' AND l_discount BETWEEN 0.05 AND 0.07 AND l_quantity < 24", - "metadata": { - "tables": ["lineitem"], - "filters": [ - {"table": "lineitem", "column": "l_shipdate", "operator": ">="}, - {"table": "lineitem", "column": "l_shipdate", "operator": "<"}, - {"table": "lineitem", "column": "l_discount", "operator": "between"}, - {"table": "lineitem", "column": "l_quantity", "operator": "<"} - ], - "aggregate": true, - "sort_columns": [] - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql b/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql deleted file mode 100644 index 80568de8..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/data/tpch_sf1/schema.sql +++ /dev/null @@ -1,95 +0,0 @@ --- TPC-H SF1 Schema (8 tables) --- Primary keys create implicit indexes tracked in raw_task.json existing_indexes - -CREATE TABLE nation ( - n_nationkey INTEGER NOT NULL, - n_name CHAR(25) NOT NULL, - n_regionkey INTEGER NOT NULL, - n_comment VARCHAR(152), - PRIMARY KEY (n_nationkey) -); - -CREATE TABLE region ( - r_regionkey INTEGER NOT NULL, - r_name CHAR(25) NOT NULL, - r_comment VARCHAR(152), - PRIMARY KEY (r_regionkey) -); - -CREATE TABLE part ( - p_partkey INTEGER NOT NULL, - p_name VARCHAR(55) NOT NULL, - p_mfgr CHAR(25) NOT NULL, - p_brand CHAR(10) NOT NULL, - p_type VARCHAR(25) NOT NULL, - p_size INTEGER NOT NULL, - p_container CHAR(10) NOT NULL, - p_retailprice DECIMAL(15,2) NOT NULL, - p_comment VARCHAR(23) NOT NULL, - PRIMARY KEY (p_partkey) -); - -CREATE TABLE supplier ( - s_suppkey INTEGER NOT NULL, - s_name CHAR(25) NOT NULL, - s_address VARCHAR(40) NOT NULL, - s_nationkey INTEGER NOT NULL, - s_phone CHAR(15) NOT NULL, - s_acctbal DECIMAL(15,2) NOT NULL, - s_comment VARCHAR(101) NOT NULL, - PRIMARY KEY (s_suppkey) -); - -CREATE TABLE partsupp ( - ps_partkey INTEGER NOT NULL, - ps_suppkey INTEGER NOT NULL, - ps_availqty INTEGER NOT NULL, - ps_supplycost DECIMAL(15,2) NOT NULL, - ps_comment VARCHAR(199) NOT NULL, - PRIMARY KEY (ps_partkey, ps_suppkey) -); - -CREATE TABLE customer ( - c_custkey INTEGER NOT NULL, - c_name VARCHAR(25) NOT NULL, - c_address VARCHAR(40) NOT NULL, - c_nationkey INTEGER NOT NULL, - c_phone CHAR(15) NOT NULL, - c_acctbal DECIMAL(15,2) NOT NULL, - c_mktsegment CHAR(10) NOT NULL, - c_comment VARCHAR(117) NOT NULL, - PRIMARY KEY (c_custkey) -); - -CREATE TABLE orders ( - o_orderkey BIGINT NOT NULL, - o_custkey INTEGER NOT NULL, - o_orderstatus CHAR(1) NOT NULL, - o_totalprice DECIMAL(15,2) NOT NULL, - o_orderdate DATE NOT NULL, - o_orderpriority CHAR(15) NOT NULL, - o_clerk CHAR(15) NOT NULL, - o_shippriority INTEGER NOT NULL, - o_comment VARCHAR(79) NOT NULL, - PRIMARY KEY (o_orderkey) -); - -CREATE TABLE lineitem ( - l_orderkey BIGINT NOT NULL, - l_partkey INTEGER NOT NULL, - l_suppkey INTEGER NOT NULL, - l_linenumber INTEGER NOT NULL, - l_quantity DECIMAL(15,2) NOT NULL, - l_extendedprice DECIMAL(15,2) NOT NULL, - l_discount DECIMAL(15,2) NOT NULL, - l_tax DECIMAL(15,2) NOT NULL, - l_returnflag CHAR(1) NOT NULL, - l_linestatus CHAR(1) NOT NULL, - l_shipdate DATE NOT NULL, - l_commitdate DATE NOT NULL, - l_receiptdate DATE NOT NULL, - l_shipinstruct CHAR(25) NOT NULL, - l_shipmode CHAR(10) NOT NULL, - l_comment VARCHAR(44) NOT NULL, - PRIMARY KEY (l_orderkey, l_linenumber) -); diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt deleted file mode 100644 index 8eb44074..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/agent_files.txt +++ /dev/null @@ -1,10 +0,0 @@ -README.md -README_zh-CN.md -Task.md -Task_zh-CN.md -scripts/init.py -verification/evaluator.py -baseline/heuristic.py -data/raw_task.json -data/tpch_sf1/queries/ -references/constants.json diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt deleted file mode 100644 index 4fe6c4c1..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/artifact_files.txt +++ /dev/null @@ -1,2 +0,0 @@ -metrics.json -outputs/*.json diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt deleted file mode 100644 index b9411b3d..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/candidate_destination.txt +++ /dev/null @@ -1 +0,0 @@ -scripts/init.py diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt deleted file mode 100644 index c4a5f13d..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/constraints.txt +++ /dev/null @@ -1,13 +0,0 @@ -IndexOptimization constraints: -1) Candidate file is `scripts/init.py` and must output JSON with: - - `indexes`: list of {"table": str, "columns": list[str], "method": "btree"} -2) SQL safety: - - Only B-tree index method is supported. - - Index columns must exist in the target table's schema. -3) Semantic correctness: - - All workload queries must return identical results before and after indexing. -4) Limits: - - max indexes = 10 - - max index storage = 500 MB -5) Read-only benchmark assets: - - verification/evaluator.py, data/, references/, baseline/. diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt deleted file mode 100644 index 613443e7..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/eval_command.txt +++ /dev/null @@ -1 +0,0 @@ -{python} {benchmark}/verification/evaluator.py {candidate} diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt deleted file mode 100644 index b9411b3d..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/initial_program.txt +++ /dev/null @@ -1 +0,0 @@ -scripts/init.py diff --git a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt b/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt deleted file mode 100644 index c4bb61c3..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/frontier_eval/readonly_files.txt +++ /dev/null @@ -1,4 +0,0 @@ -verification/evaluator.py -data/ -references/ -baseline/ diff --git a/benchmarks/ComputerSystems/IndexOptimization/references/constants.json b/benchmarks/ComputerSystems/IndexOptimization/references/constants.json deleted file mode 100644 index 3dd76b32..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/references/constants.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "scoring": { - "storage_penalty_weight": 0.3, - "count_penalty_weight": 0.1, - "numeric_tolerance": 1e-6, - "parameter_notes": { - "storage_penalty_weight": "At 50% storage ratio (max allowed), penalizes score by 0.3. Chosen so that moderate storage use is acceptable but excessive indexing is discouraged.", - "count_penalty_weight": "At max 10 indexes, penalizes score by 0.1. Lower than storage_penalty to prioritize storage cost over index count.", - "numeric_tolerance": "Standard floating-point tolerance for result correctness comparison." - } - }, - "measurement": { - "query_repeats": 3, - "warmup_rounds": 2, - "statement_timeout_s": 60, - "single_threaded": true, - "parameter_notes": { - "query_repeats": "Median of 3 runs reduces timing noise without excessive wall-clock cost.", - "warmup_rounds": "2 warmup rounds stabilize buffer pool and query planning before measurement.", - "single_threaded": "SET max_parallel_workers_per_gather = 0 ensures reproducible single-threaded timing." - } - } -} diff --git a/benchmarks/ComputerSystems/IndexOptimization/scripts/init.py b/benchmarks/ComputerSystems/IndexOptimization/scripts/init.py deleted file mode 100644 index 95848322..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/scripts/init.py +++ /dev/null @@ -1,92 +0,0 @@ -# EVOLVE-BLOCK-START -"""Index Optimization candidate program — recommend PostgreSQL indexes for TPC-H workload.""" - -from __future__ import annotations - -import argparse -import json -import math -from pathlib import Path -from typing import Any - - -# DO NOT MODIFY: CLI contract -def _parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description="Index Optimization candidate — recommend PostgreSQL indexes" - ) - parser.add_argument("--input", required=True, help="Path to raw_task.json") - parser.add_argument("--output", required=True, help="Output JSON path") - return parser.parse_args() - - -# DO NOT MODIFY: input loading -def load_json(path: str | Path) -> dict[str, Any]: - with Path(path).open() as f: - return json.load(f) - - -# DO NOT MODIFY: recursive resolve of query references -def _resolve_queries(task_dir: Path, raw: dict[str, Any]) -> list[dict[str, Any]]: - queries: list[dict[str, Any]] = [] - for ref in raw.get("queries", []): - qpath = (task_dir / ref).resolve() - q = load_json(qpath) - # Overlay benchmark_id context - q["__query_file__"] = str(qpath) - queries.append(q) - return queries - - -# MODIFIABLE: index recommendation strategy (core optimization target) -def recommend_indexes( - queries: list[dict[str, Any]], - schema: dict[str, Any], - constraints: dict[str, Any], -) -> list[dict[str, Any]]: - """ - Recommend a set of B-tree indexes to optimize query performance. - - Args: - queries: List of query dicts, each with {"id", "sql", "metadata"}. - schema: Dict mapping table name to {"columns", "row_count", "existing_indexes"}. - constraints: {"max_indexes": int, "max_storage_mb": int, ...} - - Returns: - List of index specs: [{"table": str, "columns": list[str], "method": "btree"}, ...] - """ - # Baseline: no additional indexes beyond existing primary keys - return [] - - -# DO NOT MODIFY: output format -def solve(raw: dict[str, Any], task_dir: Path) -> dict[str, Any]: - queries = _resolve_queries(task_dir, raw) - schema = raw.get("schema", {}) - constraints = raw.get("constraints", {}) - indexes = recommend_indexes(queries, schema, constraints) - return { - "indexes": indexes, - "benchmark_id": raw.get("benchmark_id", "index_optimization"), - } - - -# DO NOT MODIFY: entry point -def main() -> None: - args = _parse_args() - input_path = Path(args.input).resolve() - output_path = Path(args.output).resolve() - - task_dir = input_path.parent.parent # data/ -> task root - raw = load_json(input_path) - submission = solve(raw, task_dir) - - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text(json.dumps(submission, indent=2), encoding="utf-8") - print(f"indexes: {len(submission.get('indexes', []))}") - print(f"submission: {output_path}") - - -if __name__ == "__main__": - main() -# EVOLVE-BLOCK-END diff --git a/benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile b/benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile deleted file mode 100644 index 29a83bd7..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/verification/docker/Dockerfile +++ /dev/null @@ -1,7 +0,0 @@ -FROM postgres:16 - -# Python runtime for evaluator scripts -RUN apt-get update && apt-get install -y python3 python3-pip python3-psycopg2 && rm -rf /var/lib/apt/lists/* - -# Default entry point: PostgreSQL -CMD ["postgres"] diff --git a/benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py b/benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py deleted file mode 100644 index b11d8356..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/verification/evaluator.py +++ /dev/null @@ -1,672 +0,0 @@ -"""Evaluator for PostgreSQL index optimization benchmark (TPC-H SF1 workload).""" - -from __future__ import annotations - -import argparse -import json -import math -import os -import re -import shutil -import subprocess -import sys -import tempfile -import time -import traceback -from pathlib import Path -from typing import Any - -INVALID_COMBINED_SCORE = -1e18 - -# --------------------------------------------------------------------------- -# Path helpers -# --------------------------------------------------------------------------- - -def _is_repo_root(path: Path) -> bool: - return (path / "benchmarks").is_dir() and (path / "frontier_eval").is_dir() - - -def _find_repo_root() -> Path: - env_root = (os.environ.get("FRONTIER_ENGINEERING_ROOT") or "").strip() - if env_root: - cand = Path(env_root).expanduser().resolve() - if _is_repo_root(cand): - return cand - here = Path(__file__).resolve() - for parent in [here.parent, *here.parents]: - if _is_repo_root(parent): - return parent - return Path.cwd().resolve() - - -def _task_dir(repo_root: Path) -> Path: - return repo_root / "benchmarks" / "ComputerSystems" / "IndexOptimization" - - -# --------------------------------------------------------------------------- -# JSON helpers -# --------------------------------------------------------------------------- - -def load_json(path: str | Path) -> dict[str, Any]: - with Path(path).open() as f: - return json.load(f) - - -def dump_json(path: str | Path, payload: dict[str, Any]) -> None: - target = Path(path) - target.parent.mkdir(parents=True, exist_ok=True) - with target.open("w") as f: - json.dump(payload, f, indent=2, sort_keys=True) - - -# --------------------------------------------------------------------------- -# PostgreSQL helpers -# --------------------------------------------------------------------------- - -def _pg_connect(port: int, dbname: str = "postgres") -> Any: - import psycopg2 - return psycopg2.connect( - host="localhost", port=port, dbname=dbname, - user="postgres", password="postgres", - ) - - -def _run_sql(conn: Any, sql: str, timeout_s: int = 60) -> list[tuple]: - with conn.cursor() as cur: - cur.execute(f"SET statement_timeout = '{timeout_s}s'") - cur.execute(sql) - if cur.description is not None: - return cur.fetchall() - return [] - - -def _table_exists(conn: Any, table: str) -> bool: - sql = "SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = %s)" - with conn.cursor() as cur: - cur.execute(sql, (table,)) - return cur.fetchone()[0] - - -def _column_exists(conn: Any, table: str, column: str) -> bool: - sql = "SELECT EXISTS (SELECT FROM information_schema.columns WHERE table_name = %s AND column_name = %s)" - with conn.cursor() as cur: - cur.execute(sql, (table, column)) - return cur.fetchone()[0] - - -def _get_index_size_mb(conn: Any) -> float: - sql = """ - SELECT COALESCE(SUM(pg_relation_size(indexrelid)), 0) / 1048576.0 - FROM pg_stat_user_indexes - """ - with conn.cursor() as cur: - cur.execute(sql) - return float(cur.fetchone()[0]) - - -def _get_data_size_mb(conn: Any) -> float: - sql = """ - SELECT COALESCE(SUM(pg_relation_size(relid)), 0) / 1048576.0 - FROM pg_stat_user_tables - """ - with conn.cursor() as cur: - cur.execute(sql) - return float(cur.fetchone()[0]) - - -def _quote_ident(name: str) -> str: - return f'"{name}"' - - -def _create_single_index(conn: Any, table: str, columns: list[str], method: str) -> None: - cols = ", ".join(_quote_ident(c) for c in columns) - idx_name = f"idx_{table}_{'_'.join(columns)}" - sql = f"CREATE INDEX IF NOT EXISTS {_quote_ident(idx_name)} ON {_quote_ident(table)} USING {method} ({cols})" - with conn.cursor() as cur: - cur.execute(sql) - conn.commit() - - -def _get_actual_indexes(conn: Any) -> list[dict[str, Any]]: - """Return list of actual indexes currently in the database.""" - sql = """ - SELECT schemaname, tablename, indexname, indexdef - FROM pg_indexes - WHERE schemaname = 'public' - ORDER BY tablename, indexname - """ - with conn.cursor() as cur: - cur.execute(sql) - rows = cur.fetchall() - result = [] - for row in rows: - result.append({ - "table": row[1], - "index_name": row[2], - "definition": row[3], - }) - return result - - -# --------------------------------------------------------------------------- -# Container management -# --------------------------------------------------------------------------- - -def _start_postgres(docker_image: str, data_dir: Path | None = None) -> tuple[str, int]: - """Start a PostgreSQL Docker container, return (container_id, port).""" - cmd = [ - "docker", "run", "-d", - "--rm", - "-e", "POSTGRES_PASSWORD=postgres", - "-e", "POSTGRES_DB=postgres", - "-P", # random port - ] - if data_dir is not None and data_dir.exists(): - cmd += ["-v", f"{data_dir.resolve()}:/var/lib/postgresql/data"] - cmd.append(docker_image) - - result = subprocess.run(cmd, capture_output=True, text=True, timeout=30) - if result.returncode != 0: - raise RuntimeError(f"Docker start failed: {result.stderr}") - - container_id = result.stdout.strip() - - # Get the mapped port - port_result = subprocess.run( - ["docker", "port", container_id, "5432"], - capture_output=True, text=True, timeout=10, - ) - if port_result.returncode != 0: - _stop_postgres(container_id) - raise RuntimeError(f"Failed to get port: {port_result.stderr}") - - port = int(port_result.stdout.strip().split(":")[-1]) - - # Wait for PostgreSQL to be ready - for _ in range(30): - try: - conn = _pg_connect(port) - conn.close() - return container_id, port - except Exception: - time.sleep(1) - - _stop_postgres(container_id) - raise RuntimeError("PostgreSQL did not become ready within 30s") - - -def _stop_postgres(container_id: str) -> None: - subprocess.run(["docker", "stop", container_id], capture_output=True, timeout=30) - - -# --------------------------------------------------------------------------- -# Data restore helper -# --------------------------------------------------------------------------- - -def _restore_dump(container_id: str, dump_path: Path) -> None: - """Copy and restore a pg_dump archive into a running PostgreSQL container.""" - subprocess.run( - f"docker cp {dump_path} {container_id}:/tmp/tpch_sf1.dump".split(), - capture_output=True, timeout=30, - ) - subprocess.run( - ["docker", "exec", "-i", container_id, - "pg_restore", "-U", "postgres", "-d", "postgres", - "-Fc", "--clean", "/tmp/tpch_sf1.dump"], - capture_output=True, text=True, timeout=300, - ) - - -# --------------------------------------------------------------------------- -# SQL loading helpers -# --------------------------------------------------------------------------- - -def _resolve_queries(task_dir: Path, raw: dict[str, Any]) -> list[dict[str, Any]]: - queries = [] - for ref in raw.get("queries", []): - qpath = (task_dir / ref).resolve() - q = load_json(qpath) - queries.append(q) - return queries - - -# --------------------------------------------------------------------------- -# Result normalization & comparison -# --------------------------------------------------------------------------- - -def _normalize_value(v: Any) -> Any: - if isinstance(v, float): - return round(v, 8) - if isinstance(v, int): - return float(v) - if isinstance(v, (list, tuple)): - return tuple(_normalize_value(x) for x in v) - if isinstance(v, dict): - return {k: _normalize_value(v) for k, v in v.items()} - if v is None: - return None - if isinstance(v, bool): - return v - return str(v) - - -def _results_match(baseline_rows: list[tuple], candidate_rows: list[tuple]) -> bool: - """Compare two result sets. Handle ORDERED vs unordered comparison.""" - if len(baseline_rows) != len(candidate_rows): - return False - # Use normalized comparison - bn = [_normalize_value(r) for r in baseline_rows] - cn = [_normalize_value(r) for r in candidate_rows] - # Try ordered comparison first, fall back to sorted - if bn == cn: - return True - return sorted(bn) == sorted(cn) - - -# --------------------------------------------------------------------------- -# Timing -# --------------------------------------------------------------------------- - -def _time_query(conn: Any, sql: str, repeats: int = 3, timeout_s: int = 60) -> float: - """Execute a query `repeats` times and return the median time in seconds.""" - times = [] - for _ in range(repeats): - start = time.time() - try: - _run_sql(conn, sql, timeout_s) - except Exception as e: - raise RuntimeError(f"Query failed: {e}") from e - elapsed = time.time() - start - times.append(elapsed) - times.sort() - return times[len(times) // 2] - - -# --------------------------------------------------------------------------- -# Validation -# --------------------------------------------------------------------------- - -def _validate_index_def( - idx: dict[str, Any], conn: Any, -) -> tuple[bool, str]: - table = idx.get("table", "") - columns = idx.get("columns", []) - method = idx.get("method", "") - - if not table: - return False, "missing table name" - if not columns or not isinstance(columns, list): - return False, "columns must be a non-empty list" - if method != "btree": - return False, f"unsupported index method: {method}" - - if not _table_exists(conn, table): - return False, f"table does not exist: {table}" - - for col in columns: - if not _column_exists(conn, table, col): - return False, f"column does not exist: {table}.{col}" - - return True, "" - - -# --------------------------------------------------------------------------- -# Submission runner -# --------------------------------------------------------------------------- - -def _run_submission(script_path: Path, input_path: Path, output_path: Path, cwd: Path) -> tuple[dict[str, Any], str, str]: - """Run a submission script and return parsed output.""" - result = subprocess.run( - [sys.executable, str(script_path), "--input", str(input_path), "--output", str(output_path)], - capture_output=True, text=True, timeout=60, cwd=str(cwd), - ) - if result.returncode != 0: - raise RuntimeError(f"Script failed (exit {result.returncode}): {result.stderr[:500]}") - - if not output_path.exists(): - raise RuntimeError(f"Output file not created: {output_path}") - - try: - submission = load_json(output_path) - except Exception as e: - raise RuntimeError(f"Invalid output JSON: {e}") from e - - return submission, result.stdout, result.stderr - - -# --------------------------------------------------------------------------- -# Score computation -# --------------------------------------------------------------------------- - -def _compute_metrics( - no_index_time: float, - baseline_time: float, - candidate_time: float, - n_indexes: int, - storage_mb: float, - data_size_mb: float, - correct: bool, - warmup_ok: bool, -) -> dict[str, Any]: - """Compute all metrics and validation flags.""" - - metrics: dict[str, Any] = { - "valid": 0.0, - "combined_score": INVALID_COMBINED_SCORE, - "no_index_time_s": round(no_index_time, 4), - "baseline_time_s": round(baseline_time, 4), - "candidate_time_s": round(candidate_time, 4), - "n_indexes": n_indexes, - "storage_mb": round(storage_mb, 2), - "data_size_mb": round(data_size_mb, 2), - "correct": 1.0 if correct else 0.0, - } - - # Hard constraints - valid = ( - correct - and warmup_ok - and n_indexes <= 10 - and storage_mb <= 500.0 - ) - - if not valid: - metrics["valid"] = 0.0 - metrics["combined_score"] = INVALID_COMBINED_SCORE - return metrics - - speedup = baseline_time / max(candidate_time, 1e-9) - metrics["speedup"] = round(speedup, 4) - - log_speedup = math.log2(max(speedup, 1e-6)) - storage_penalty = 0.3 * (storage_mb / 500.0) - count_penalty = 0.1 * (n_indexes / 10.0) - combined = log_speedup * (1.0 - storage_penalty - count_penalty) - - metrics["valid"] = 1.0 - metrics["combined_score"] = round(combined, 6) - metrics["log_speedup"] = round(log_speedup, 6) - metrics["storage_penalty"] = round(storage_penalty, 6) - metrics["count_penalty"] = round(count_penalty, 6) - - return metrics - - -# --------------------------------------------------------------------------- -# Main evaluation -# --------------------------------------------------------------------------- - -def evaluate( - program_path: str, - *, - timeout_s: float = 300.0, - repo_root: Path | None = None, -) -> Any: - """ - UnifiedTask evaluator interface. - - 1. Load config and generate submissions - 2. Start PostgreSQL, restore data - 3. Measure no-index, heuristic, and candidate performance - 4. Verify correctness - 5. Compute score - """ - artifacts: dict[str, Any] = {} - start_time = time.time() - - # Resolve paths - if repo_root is None: - repo_root = _find_repo_root() - task_dir = _task_dir(repo_root) - docker_image = "frontier-pg-index:latest" - raw_task_path = task_dir / "data" / "raw_task.json" - temp_dir = Path(tempfile.mkdtemp(prefix="fe_idxopt_")) - outputs_dir = task_dir / "outputs" - outputs_dir.mkdir(exist_ok=True) - - try: - # ------------------------------------------------------------------ - # Phase A: Generate submissions - # ------------------------------------------------------------------ - raw = load_json(raw_task_path) - queries = _resolve_queries(task_dir, raw) - schema = raw.get("schema", {}) - constraints = raw.get("constraints", {}) - artifacts["n_queries"] = len(queries) - artifacts["query_ids"] = [q.get("id", f"q{i}") for i, q in enumerate(queries)] - - # Run baseline (heuristic) - baseline_script = task_dir / "baseline" / "heuristic.py" - baseline_output = outputs_dir / "baseline_submission.json" - try: - baseline_sub, baseline_stdout, baseline_stderr = _run_submission( - baseline_script, raw_task_path, baseline_output, task_dir, - ) - artifacts["baseline_stdout"] = baseline_stdout[:2000] - except Exception as e: - artifacts["error_message"] = f"baseline failed: {e}" - return _wrap(_compute_metrics(0, 0, 0, 0, 0, 0, False, False), artifacts) - - # Run candidate - candidate_script = Path(program_path).resolve() - candidate_output = outputs_dir / "candidate_submission.json" - try: - candidate_sub, candidate_stdout, _ = _run_submission( - candidate_script, raw_task_path, candidate_output, task_dir, - ) - artifacts["candidate_stdout"] = candidate_stdout[:2000] - except Exception as e: - artifacts["error_message"] = f"candidate failed: {e}" - return _wrap(_compute_metrics(0, 0, 0, 0, 0, 0, False, False), artifacts) - - candidate_indexes = candidate_sub.get("indexes", []) - artifacts["candidate_n_indexes"] = len(candidate_indexes) - artifacts["candidate_indexes"] = candidate_indexes - - # ------------------------------------------------------------------ - # Phase B: Database evaluation - # ------------------------------------------------------------------ - container_id = None - port = None - - try: - container_id, port = _start_postgres(docker_image) - artifacts["postgres_port"] = port - conn = _pg_connect(port) - - # Restore data from pre-generated dump - dump_path = task_dir / "data" / "tpch_sf1" / "tpch_sf1.dump" - if not dump_path.exists(): - raise RuntimeError("no tpch_sf1.dump found — run data/tpch_sf1/gen_data.sh first") - _restore_dump(container_id, dump_path) - conn = _pg_connect(port) - - # Verify existing indexes in database match metadata - actual_idxs = _get_actual_indexes(conn) - artifacts["actual_indexes"] = actual_idxs - if len(actual_idxs) == 0: - raise RuntimeError("No primary key indexes created - schema may not be loaded correctly") - - # Prepare query list for timing - query_sqls = [q.get("sql", "") for q in queries] - query_ids = [q.get("id", f"q{i}") for i, q in enumerate(queries)] - - # -------------------------------------------------------------- - # Measure no-index time (reference only) - # -------------------------------------------------------------- - no_index_times = [] - for sql in query_sqls: - t = _time_query(conn, sql, repeats=3) - no_index_times.append(t) - no_index_total = sum(no_index_times) - artifacts["no_index_query_times"] = no_index_times - - # -------------------------------------------------------------- - # Measure heuristic time (baseline) - # -------------------------------------------------------------- - baseline_indexes = baseline_sub.get("indexes", []) - for idx in baseline_indexes: - ok, msg = _validate_index_def(idx, conn) - if not ok: - raise RuntimeError(f"Invalid baseline index: {msg}") - _create_single_index(conn, idx["table"], idx["columns"], idx["method"]) - conn.commit() - - # Warmup - for sql in query_sqls: - try: - _run_sql(conn, sql, timeout_s=30) - except Exception: - pass - - baseline_times = [] - for sql in query_sqls: - t = _time_query(conn, sql, repeats=3) - baseline_times.append(t) - baseline_total = sum(baseline_times) - artifacts["baseline_query_times"] = baseline_times - artifacts["baseline_indexes"] = baseline_indexes - - # Store baseline results for correctness comparison - baseline_results = [] - for sql in query_sqls: - baseline_results.append(_run_sql(conn, sql)) - conn.close() - - # -------------------------------------------------------------- - # Measure candidate time - # -------------------------------------------------------------- - _stop_postgres(container_id) - container_id = None - - # Fresh instance for candidate - container_id2, port2 = _start_postgres(docker_image) - conn2 = _pg_connect(port2) - _restore_dump(container_id2, dump_path) - conn2 = _pg_connect(port2) - - # Validate and create candidate indexes - for idx in candidate_indexes: - ok, msg = _validate_index_def(idx, conn2) - if not ok: - raise RuntimeError(f"Invalid candidate index: {msg}") - _create_single_index(conn2, idx["table"], idx["columns"], idx["method"]) - conn2.commit() - - # Measure storage - storage_mb = _get_index_size_mb(conn2) - data_size_mb = _get_data_size_mb(conn2) - - # Warmup - for sql in query_sqls: - try: - _run_sql(conn2, sql, timeout_s=30) - except Exception: - pass - - candidate_times = [] - for sql in query_sqls: - t = _time_query(conn2, sql, repeats=3) - candidate_times.append(t) - candidate_total = sum(candidate_times) - artifacts["candidate_query_times"] = candidate_times - - # Correctness verification - candidate_results = [] - for sql in query_sqls: - candidate_results.append(_run_sql(conn2, sql)) - - all_correct = True - mismatches = [] - for i, (br, cr) in enumerate(zip(baseline_results, candidate_results)): - if not _results_match(br, cr): - all_correct = False - mismatches.append(query_ids[i]) - artifacts["mismatch_queries"] = mismatches - - conn2.close() - _stop_postgres(container_id2) - - # Compute final score - metrics = _compute_metrics( - no_index_total, - baseline_total, - candidate_total, - len(candidate_indexes), - storage_mb, - data_size_mb, - all_correct, - True, - ) - metrics["runtime_s"] = round(time.time() - start_time, 4) - return _wrap(metrics, artifacts) - - except Exception as e: - traceback.print_exc() - artifacts["error_message"] = str(e)[:500] - err_metrics = { - "valid": 0.0, - "combined_score": INVALID_COMBINED_SCORE, - "no_index_time_s": 0.0, - "baseline_time_s": 0.0, - "candidate_time_s": 0.0, - "n_indexes": 0, - "storage_mb": 0.0, - "data_size_mb": 0.0, - "correct": 0.0, - "timeout": 1.0 if "timeout" in str(e).lower() else 0.0, - "runtime_s": round(time.time() - start_time, 4), - } - return _wrap(err_metrics, artifacts) - finally: - if container_id is not None: - try: - _stop_postgres(container_id) - except Exception: - pass - - except Exception as e: - traceback.print_exc() - artifacts["error_message"] = str(e)[:500] - return _wrap( - {"valid": 0.0, "combined_score": INVALID_COMBINED_SCORE, "runtime_s": round(time.time() - start_time, 4)}, - artifacts, - ) - finally: - shutil.rmtree(temp_dir, ignore_errors=True) - - -def _wrap(metrics: dict[str, Any], artifacts: dict[str, Any]) -> Any: - try: - from openevolve.evaluation_result import EvaluationResult - return EvaluationResult(metrics=metrics, artifacts=artifacts) - except ImportError: - return metrics - - -# --------------------------------------------------------------------------- -# CLI -# --------------------------------------------------------------------------- - -def main() -> int: - if len(sys.argv) == 2 and not sys.argv[1].startswith("--"): - result = evaluate(sys.argv[1]) - metrics = result.metrics if hasattr(result, "metrics") else result - print(json.dumps(metrics)) - return 0 if metrics.get("valid", 0) > 0 else 1 - - parser = argparse.ArgumentParser(description="IndexOptimization evaluator") - parser.add_argument("program", help="Path to candidate program") - parser.add_argument("--timeout", type=float, default=300.0) - parser.add_argument("--repo-root", default=None) - args = parser.parse_args() - - repo_root = Path(args.repo_root).resolve() if args.repo_root else None - result = evaluate(args.program, timeout_s=args.timeout, repo_root=repo_root) - metrics = result.metrics if hasattr(result, "metrics") else result - print(json.dumps(metrics)) - return 0 if metrics.get("valid", 0) > 0 else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt b/benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt deleted file mode 100644 index 83ec1256..00000000 --- a/benchmarks/ComputerSystems/IndexOptimization/verification/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -psycopg2-binary>=2.9 diff --git a/benchmarks/ComputerSystems/README.md b/benchmarks/ComputerSystems/README.md index 411962ac..8ad7b593 100644 --- a/benchmarks/ComputerSystems/README.md +++ b/benchmarks/ComputerSystems/README.md @@ -3,5 +3,6 @@ Includes computer-systems engineering optimization tasks: - `MallocLab`: dynamic memory allocation. - `DuckDBWorkloadOptimization`: analytical SQL workload tuning (index/materialized-view selection + query rewrite). +- `CacheReplacement`: CPU cache replacement policy optimization. (Replaces the deleted `IndexOptimization`, which overlapped with `DuckDBWorkloadOptimization`.) Note for contributors: ensure the evolved baseline source file contains `EVOLVE-BLOCK-START` / `EVOLVE-BLOCK-END` markers (use `// ...` in C/C++). diff --git a/benchmarks/ComputerSystems/README_zh-CN.md b/benchmarks/ComputerSystems/README_zh-CN.md index 0d12dfc1..364d6188 100644 --- a/benchmarks/ComputerSystems/README_zh-CN.md +++ b/benchmarks/ComputerSystems/README_zh-CN.md @@ -3,5 +3,6 @@ 包含以下计算机系统工程优化任务： - `MallocLab`：动态内存分配。 - `DuckDBWorkloadOptimization`：分析型 SQL 负载调优（索引/物化视图选择 + 查询改写）。 +- `CacheReplacement`：CPU 缓存替换策略优化。（替代已删除的 `IndexOptimization`，因其与 `DuckDBWorkloadOptimization` 功能重叠。） 贡献提示：请确保被 evolve 的 baseline 源码文件包含 `EVOLVE-BLOCK-START` / `EVOLVE-BLOCK-END` 标记（C/C++ 中使用 `// ...`）。