From ef16970fd98e4c4583ed6c86cad88545b22fbaf6 Mon Sep 17 00:00:00 2001 From: Daniele Lotito Date: Mon, 16 Mar 2026 10:31:48 +0000 Subject: [PATCH 1/6] docs(skills): avoid platform specific commands and clarify output format option --- .cursor/skills/using-ado-cli/SKILL.md | 37 +++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/.cursor/skills/using-ado-cli/SKILL.md b/.cursor/skills/using-ado-cli/SKILL.md index 635b0546e..c2ca3e5c0 100644 --- a/.cursor/skills/using-ado-cli/SKILL.md +++ b/.cursor/skills/using-ado-cli/SKILL.md @@ -29,6 +29,43 @@ uv run ado [COMMAND] [SUBCOMMAND1] [SUBCOMMAND2] --help - Required arguments are included - Optional flags match actual CLI behavior +## Common Option Mistakes + +### Output Format Options + +Different commands use different flags for output format. **Always verify with `--help`**. + +| Command | Correct Flag | ❌ Common Mistakes | +| ------- | ----------- | ----------------- | +| `ado get` | `--output` or `-o` | `--output-format`, `--format` | +| `ado show entities` | `--output-format` | `--format`, `--output`, `-o` | +| `ado show requests` | `--output-format` or `-o` | `--format` | +| `ado show results` | `--output-format` or `-o` | `--format` | + +**Examples:** + +```bash +# ✅ Correct +uv run ado get operations --output json +uv run ado get operations -o json +uv run ado show entities space SPACE_ID --output-format csv + +# ❌ Wrong - will fail +uv run ado get operations --output-format json +uv run ado show entities space SPACE_ID --format csv +uv run ado show entities space SPACE_ID --output csv +``` + +### Platform-Specific Issues + +When writing scripts that use grep or other shell commands: + +**Example with grep patterns:** + +- ❌ Don't use: `grep -P` (Perl regex, not available on macOS) +- ✅ Use: `grep -E` (extended regex, cross-platform) +- ✅ Or: basic grep patterns without flags + ## Commands That do not exist These plausible-sounding commands do not exist in ado. Do not write them: From 54567bc1c279a02eaa2f82a40efca3e0bbd70c12 Mon Sep 17 00:00:00 2001 From: Daniele Lotito Date: Mon, 16 Mar 2026 13:10:28 +0000 Subject: [PATCH 2/6] docs(skill): add custom experiment skill reference --- .../skills/create-custom-experiment/SKILL.md | 327 ++++++++++++++++++ 1 file changed, 327 insertions(+) create mode 100644 .cursor/skills/create-custom-experiment/SKILL.md diff --git a/.cursor/skills/create-custom-experiment/SKILL.md b/.cursor/skills/create-custom-experiment/SKILL.md new file mode 100644 index 000000000..c5f83a952 --- /dev/null +++ b/.cursor/skills/create-custom-experiment/SKILL.md @@ -0,0 +1,327 @@ +--- +name: create-custom-experiment +description: Create and test custom experiments in ado. Covers package structure, decorator usage, installation, and testing via run_experiment API and discoveryspace dry-run. +--- + +# Create Custom Experiment + +Custom experiments enable using Python functions as experiments in ado by +registering them with the `@custom_experiment` decorator. + +## Package Structure + +Standard Python package with entry-point registration: + +```text +$YOUR_REPO_NAME/ + pyproject.toml + my_custom_experiment/ + __init__.py + experiments.py # Decorated function(s) +``` + +## pyproject.toml Configuration + +Required entry-point in `pyproject.toml`: + +```toml +[project] +name = "my_custom_experiment" +description = "Custom experiment description" +dependencies = [] # Add any dependencies +dynamic = ["version"] + +[project.entry-points."ado.custom_experiments"] +my_experiment = "my_custom_experiment.experiments" + +[build-system] +requires = ["hatchling", "uv-dynamic-versioning>=0.7.0"] +build-backend = "hatchling.build" + +[tool.hatch.build] +include = ["my_custom_experiment/**"] + +[tool.hatch.build.targets.wheel] +only-include = ["my_custom_experiment"] + +[tool.hatch.version] +source = "uv-dynamic-versioning" + +[tool.uv-dynamic-versioning] +vcs = "git" +style = "pep440" +pattern = "default-unprefixed" +fallback-version = "0.0.0" +tagged-metadata = true +dirty = true +bump = true +``` + +## Decorator Usage + +### Simple Example + +```python +from typing import Any +from orchestrator.modules.actuators.custom_experiments import custom_experiment + +@custom_experiment(output_property_identifiers=["density"]) +def calculate_density(mass: float, volume: float) -> dict[str, Any]: + density_value = mass / volume if volume else None + return {"density": density_value} +``` + +**Key points:** + +- Function name becomes experiment identifier +- Positional parameters → required properties +- Return dict with keys matching `output_property_identifiers` +- Type annotations determine domain inference: + - `float` → continuous domain + - `int` → discrete domain + - `Literal` → categorical domain + +### Advanced Example with Explicit Domains + +```python +from orchestrator.modules.actuators.custom_experiments import custom_experiment +from orchestrator.schema.domain import PropertyDomain, VariableTypeEnum +from orchestrator.schema.property import ConstitutiveProperty + +@custom_experiment( + required_properties=[ + ConstitutiveProperty( + identifier="x0", + propertyDomain=PropertyDomain( + variableType=VariableTypeEnum.CONTINUOUS_VARIABLE_TYPE, + domainRange=[-5, 5] + ) + ), + ConstitutiveProperty( + identifier="x1", + propertyDomain=PropertyDomain( + variableType=VariableTypeEnum.CONTINUOUS_VARIABLE_TYPE, + domainRange=[-5, 5] + ) + ) + ], + optional_properties=[ + ConstitutiveProperty( + identifier="num_iterations", + propertyDomain=PropertyDomain( + variableType=VariableTypeEnum.DISCRETE_VARIABLE_TYPE, + domainRange=[1, 100], + interval=1 + ) + ) + ], + parameterization={"num_iterations": 10}, + output_property_identifiers=["result"], + metadata={"description": "Example optimization function"} +) +def my_experiment(x0: float, x1: float, num_iterations: int) -> dict[str, Any]: + # Implementation + result = x0**2 + x1**2 + return {"result": result} +``` + +## Installation + +Install in editable mode from package root: + +```bash +# From the directory containing pyproject.toml +uv pip install -e . +``` + +Verify installation: + +```bash +uv run ado get experiments +``` + +Custom experiment appears under `custom_experiments` actuator. + +## Testing + +### Method 1: run_experiment API + +Create test point YAML (`point.yaml`): + +```yaml +entity: + mass: 10.0 + volume: 2.0 +experiments: + - actuatorIdentifier: custom_experiments + experimentIdentifier: calculate_density +``` + +Execute: + +```bash +uv run run_experiment point.yaml +``` + +**Options:** + +- `--remote `: Execute via REST API +- `--timeout `: Timeout for remote execution (default: 300) +- `--no-validate`: Skip entity validation +- `--actuator-configuration-id `: Specify actuator configuration + +### Method 2: discoveryspace Dry-Run + +Create space YAML (`space.yaml`): + +```yaml +entitySpace: + - identifier: mass + propertyDomain: + variableType: CONTINUOUS_VARIABLE_TYPE + domainRange: [1, 10] + - identifier: volume + propertyDomain: + variableType: CONTINUOUS_VARIABLE_TYPE + domainRange: [1, 10] +experiments: + - actuatorIdentifier: custom_experiments + experimentIdentifier: calculate_density +``` + +**Important:** In space YAML files, `variableType` must use full enum +constant names: + +- `CONTINUOUS_VARIABLE_TYPE` (for float ranges) +- `DISCRETE_VARIABLE_TYPE` (for integer ranges with interval) +- `CATEGORICAL_VARIABLE_TYPE` (for fixed set of values) +- `BINARY_VARIABLE_TYPE` (for true/false values) + +Do not use shortened forms like `continuous`, `discrete`, or `categorical` - +these will cause validation errors. + +Test with dry-run: + +```bash +uv run ado create space -f space.yaml --dry-run +``` + +Validates: + +- Space configuration +- Experiment availability +- Property domain compatibility + +## Validation Steps + +1. **Verify experiment registration:** + + ```bash + uv run ado get experiments + ``` + +2. **Check experiment details:** + + ```bash + uv run ado get experiments --details + ``` + +3. **Test with point:** + + ```bash + uv run run_experiment point.yaml + ``` + +4. **Validate space configuration:** + + ```bash + uv run ado create space -f space.yaml --dry-run + ``` + +## Common Patterns + +### Multiple Experiments in One Module + +```python +@custom_experiment(output_property_identifiers=["result1"]) +def experiment_one(x: float) -> dict[str, Any]: + return {"result1": x * 2} + +@custom_experiment(output_property_identifiers=["result2"]) +def experiment_two(y: float) -> dict[str, Any]: + return {"result2": y ** 2} +``` + +### Multiple Modules + +Register each module separately: + +```toml +[project.entry-points."ado.custom_experiments"] +module_one = "my_package.experiments_one" +module_two = "my_package.experiments_two" +``` + +### Ray Configuration + +```python +@custom_experiment( + output_property_identifiers=["loss"], + use_ray=True, + ray_options={ + "num_cpus": 2, + "num_gpus": 0.5, + "runtime_env": {"env_vars": {"OMP_NUM_THREADS": "2"}} + } +) +def heavy_experiment(x: float) -> dict[str, Any]: + # Implementation + pass +``` + +## Troubleshooting + +### Experiment Not Found + +```bash +# Reinstall package +uv pip install -e . --force-reinstall + +# Verify entry-point +uv run ado get experiments +``` + +### Domain Inference Issues + +Define domains explicitly using `required_properties` parameter. + +### Validation Errors + +Check: + +- Return dict keys match `output_property_identifiers` +- Parameter types match domain definitions +- All required parameters provided in test point + +### Domain Compatibility Errors + +When `ado create space --dry-run` fails with domain compatibility: + +- Entity space values must be subset of experiment's required property + domain +- For categorical properties, ensure all entity space values are in + experiment's allowed values +- Check experiment definition for exact domain constraints + +## References + +- [AGENTS.md](../../../AGENTS.md) - Development guidelines +- [creating-custom-experiments.md](../../../website/docs/actuators/creating-custom-experiments.md) + \- Full documentation +- [run_experiment.md](../../../website/docs/actuators/run_experiment.md) + \- run_experiment details +- [density_example/](../../../examples/density_example/) - Simple example +- [optimization_test_functions/](../../../examples/optimization_test_functions/) + \- Advanced example +- [examples/gp_tabarena_test/](../../../examples/gp_tabarena_test/) - + GP regression with TabArena datasets From 7c6fe82051d1ded3cccf61e399c16359cb7b0726 Mon Sep 17 00:00:00 2001 From: Daniele Lotito Date: Mon, 16 Mar 2026 10:41:18 +0000 Subject: [PATCH 3/6] docs(skills): add troubleshooting This might contain duplicate information but having the skill here as well reduced the error rate --- .cursor/skills/query-ado-data/SKILL.md | 29 ++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/.cursor/skills/query-ado-data/SKILL.md b/.cursor/skills/query-ado-data/SKILL.md index 970f0c6b8..659c7580c 100644 --- a/.cursor/skills/query-ado-data/SKILL.md +++ b/.cursor/skills/query-ado-data/SKILL.md @@ -284,6 +284,35 @@ uv run ado show related space SPACE_ID ## Advanced Filtering +## Troubleshooting + +### Output Format Flag Confusion + +Different commands use different flags for output format: + +- `ado get`: Use `--output` or `-o` +- `ado show entities`: Use `--output-format` +- `ado show requests`: Use `--output-format` or `-o` +- `ado show results`: Use `--output-format` or `-o` + +**Always verify with `--help` before using.** + +### Common Mistakes + +1. **Using wrong output flag:** + + ```bash + # ❌ Wrong + uv run ado get operations --output-format csv + + # ✅ Correct + uv run ado get operations --output csv + ``` + +2. **Forgetting to verify command syntax:** + - Always run `uv run ado [COMMAND] --help` before writing documentation + - Don't assume flags based on other commands + The metastore class can provide more powerful querying via scripts. See orchestrator/metastore/sqlstore.py From 8d22234855dcd3e0612e2e728f7b67e6e8cb68b5 Mon Sep 17 00:00:00 2001 From: Daniele Lotito Date: Mon, 16 Mar 2026 18:23:00 +0000 Subject: [PATCH 4/6] docs: clarify output format behavior in ado CLI skill Add explicit note that json, yaml, and raw formats print complete resource details --- .cursor/skills/using-ado-cli/SKILL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.cursor/skills/using-ado-cli/SKILL.md b/.cursor/skills/using-ado-cli/SKILL.md index c2ca3e5c0..c281dbc16 100644 --- a/.cursor/skills/using-ado-cli/SKILL.md +++ b/.cursor/skills/using-ado-cli/SKILL.md @@ -33,7 +33,9 @@ uv run ado [COMMAND] [SUBCOMMAND1] [SUBCOMMAND2] --help ### Output Format Options -Different commands use different flags for output format. **Always verify with `--help`**. +Different commands use different flags for output format. **Always verify with +`--help`**. To print the complete resource details, use `json`, `yaml`, or +`raw` formats. | Command | Correct Flag | ❌ Common Mistakes | | ------- | ----------- | ----------------- | From 31e6f118c732623b7405b83ea0baad8b4ae35b82 Mon Sep 17 00:00:00 2001 From: Daniele Lotito Date: Mon, 16 Mar 2026 18:25:08 +0000 Subject: [PATCH 5/6] docs: add command order section to common mistakes Clarify that correct order is 'ado [COMMAND] [RESOURCE_TYPE]' not the reverse --- .cursor/skills/using-ado-cli/SKILL.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/.cursor/skills/using-ado-cli/SKILL.md b/.cursor/skills/using-ado-cli/SKILL.md index c281dbc16..5932ad223 100644 --- a/.cursor/skills/using-ado-cli/SKILL.md +++ b/.cursor/skills/using-ado-cli/SKILL.md @@ -58,6 +58,22 @@ uv run ado show entities space SPACE_ID --format csv uv run ado show entities space SPACE_ID --output csv ``` +### Command Order + +The correct order is `ado [COMMAND] [RESOURCE_TYPE]`, not the reverse. + +```bash +# ✅ Correct +uv run ado get operation OPERATION_ID +uv run ado get operations +uv run ado create space -f space.yaml + +# ❌ Wrong - will fail +uv run ado operation get OPERATION_ID +uv run ado operations get +uv run ado space create -f space.yaml +``` + ### Platform-Specific Issues When writing scripts that use grep or other shell commands: From 02a78fccd5ad1f054dff0ab8424bdbe0ff96f706 Mon Sep 17 00:00:00 2001 From: Daniele Lotito Date: Fri, 20 Mar 2026 11:50:15 +0000 Subject: [PATCH 6/6] docs: update agent guidelines and skills documentation - Enhanced AGENTS.md with comprehensive development guidelines - Updated query-ado-data skill documentation - Updated using-ado-cli skill documentation --- .cursor/skills/query-ado-data/SKILL.md | 31 ++------------------------ .cursor/skills/using-ado-cli/SKILL.md | 10 --------- AGENTS.md | 5 +++++ 3 files changed, 7 insertions(+), 39 deletions(-) diff --git a/.cursor/skills/query-ado-data/SKILL.md b/.cursor/skills/query-ado-data/SKILL.md index 659c7580c..da15c30d4 100644 --- a/.cursor/skills/query-ado-data/SKILL.md +++ b/.cursor/skills/query-ado-data/SKILL.md @@ -284,38 +284,11 @@ uv run ado show related space SPACE_ID ## Advanced Filtering -## Troubleshooting - -### Output Format Flag Confusion - -Different commands use different flags for output format: - -- `ado get`: Use `--output` or `-o` -- `ado show entities`: Use `--output-format` -- `ado show requests`: Use `--output-format` or `-o` -- `ado show results`: Use `--output-format` or `-o` - -**Always verify with `--help` before using.** - -### Common Mistakes - -1. **Using wrong output flag:** - - ```bash - # ❌ Wrong - uv run ado get operations --output-format csv - - # ✅ Correct - uv run ado get operations --output csv - ``` - -2. **Forgetting to verify command syntax:** - - Always run `uv run ado [COMMAND] --help` before writing documentation - - Don't assume flags based on other commands - The metastore class can provide more powerful querying via scripts. See orchestrator/metastore/sqlstore.py +For output format flag reference, see [using-ado-cli](../using-ado-cli/). + ## References When modifying or creating code while using this skill, follow: diff --git a/.cursor/skills/using-ado-cli/SKILL.md b/.cursor/skills/using-ado-cli/SKILL.md index 5932ad223..3bef4baae 100644 --- a/.cursor/skills/using-ado-cli/SKILL.md +++ b/.cursor/skills/using-ado-cli/SKILL.md @@ -74,16 +74,6 @@ uv run ado operations get uv run ado space create -f space.yaml ``` -### Platform-Specific Issues - -When writing scripts that use grep or other shell commands: - -**Example with grep patterns:** - -- ❌ Don't use: `grep -P` (Perl regex, not available on macOS) -- ✅ Use: `grep -E` (extended regex, cross-platform) -- ✅ Or: basic grep patterns without flags - ## Commands That do not exist These plausible-sounding commands do not exist in ado. Do not write them: diff --git a/AGENTS.md b/AGENTS.md index 1e20f9905..d613d3fb7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,6 +88,11 @@ Use Test Driven Development - Run pytest: check tests - Iterate until tests pass +### Shell Commands + +- Use `rg` (ripgrep) instead of `grep` for searching files +- Avoid platform-specific flags (e.g., `grep -P` is not available on macOS) + ### Writing Tests - Check for existing fixtures before creating new ones: