From 232081249a0fd56add9fe721124498e6991e9bac Mon Sep 17 00:00:00 2001 From: tankdonut <206458089+tankdonut@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:40:06 +0000 Subject: [PATCH] feat(spec): merge flag unions config list entries with stored values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A spec config entry with an exact list value reverts any external additions on the next boot: openclaw doctor expands plugins.allow (memory-core, github) and reconcile resets it to the spec's list, flip-flopping every cycle and evicting plugins the doctor just added. merge: true (list values only, post split_csv; anything else fails closed at load) changes the apply to a union — current order first, spec additions in spec order — and skips the CLI when every spec item is already stored. A missing or non-list stored value converges to the spec list; an unreadable config skips the drift check like config_set. Failures warn naming the key only. Applied via config_merge_list ahead of the strict-json branch in reconcile_config; if_env guards unchanged. Docs updated in standard-agent.md (keys table, config-entries section, drift note). --- container/AGENTS.md | 2 +- container/entrypoint.py | 31 ++++++++- container/spec.py | 13 +++- container/test_entrypoint.py | 118 +++++++++++++++++++++++++++++++++++ container/test_spec.py | 43 +++++++++++++ docs/standard-agent.md | 13 +++- 6 files changed, 215 insertions(+), 5 deletions(-) diff --git a/container/AGENTS.md b/container/AGENTS.md index 3b98f1e..c809dcb 100644 --- a/container/AGENTS.md +++ b/container/AGENTS.md @@ -21,7 +21,7 @@ internals only. | Load | `load_agent_spec` :153 | fail-closed; `AGENT_SPEC_PATH` override | | Upgrade backup | `backup_before_upgrade` :1566 | on `AGENT_BASE_VERSION` delta vs `{data}/last-image-version`, warm volume → raw state-db snapshot into the backups dir, `openclaw doctor --fix --non-interactive` (schema migrations; the 2026.10+ CLI gates backup + gateway on them), then `openclaw backup create --verify --output /backups` (CLI refuses inside `{data}`); any failure aborts (exit 1, marker kept → retry); fresh volume records marker only | | First boot | `first_boot_setup` :277 | only when `openclaw.json` absent; llama-cpp provider install is base behavior, not spec-configurable; a failed `setup` aborts the boot cleanly (exit 1, named env var) — `zai-coding-*` auth is load-gated on `ZAI_API_KEY` (see `spec.required_env_for_auth_choice`); ends with `_snapshot_base_plugins` → `{data}/agent-managed-plugins` (ownership for the orphan report AND the plugins.allow seed) | -| Reconcile | `reconcile_config` :367 → `reconcile_mcp` :569 → `reconcile_plugins` :710 | idempotent; `config_set` shells out only on drift (compares `openclaw.json` directly), `config_set_batch` applies multi-key deltas in one `--batch-json` call; `_seed_plugins_allow` seeds the allowlist when neither config nor an env-active spec entry owns it; `_seed_agents_default_models` merges spec-referenced models (model.fallback, automations.model) + the GLM 5.3 series (`zai-coding-*` specs) into an existing `agents.defaults.models` allowlist — never creates one (absent = allow-any), never rewrites a non-map value (warns), stands down when a spec entry owns the path; `features.gateway_auth` arms auth via the env var natively and `_retire_legacy_gateway_auth_pair` unsets the legacy pair (exact-value match only); MCP entries re-register on resolved-flag drift (args-digest marker `{data}/agent-mcp-args`; hand-registered or pre-marker servers converge once; digests only, never resolved values; remote `transport` participates in the digest); MCP removal is gated on `features.mcp_prune` (ownership-marked via `{data}/agent-managed-mcp`; if_env-skipped entries count as still spec'd); MCP orphan report is warn-only; plugin orphan report is warn-only and disabled without the plugins marker | +| Reconcile | `reconcile_config` :367 → `reconcile_mcp` :569 → `reconcile_plugins` :710 | idempotent; `config_set` shells out only on drift (compares `openclaw.json` directly), `config_set_batch` applies multi-key deltas in one `--batch-json` call; `merge`-flagged list entries union with the stored list via `config_merge_list` (subset containment is the skip test, so externally added items — doctor allowlist expansions — survive reconcile); `_seed_plugins_allow` seeds the allowlist when neither config nor an env-active spec entry owns it; `_seed_agents_default_models` merges spec-referenced models (model.fallback, automations.model) + the GLM 5.3 series (`zai-coding-*` specs) into an existing `agents.defaults.models` allowlist — never creates one (absent = allow-any), never rewrites a non-map value (warns), stands down when a spec entry owns the path; `features.gateway_auth` arms auth via the env var natively and `_retire_legacy_gateway_auth_pair` unsets the legacy pair (exact-value match only); MCP entries re-register on resolved-flag drift (args-digest marker `{data}/agent-mcp-args`; hand-registered or pre-marker servers converge once; digests only, never resolved values; remote `transport` participates in the digest); MCP removal is gated on `features.mcp_prune` (ownership-marked via `{data}/agent-managed-mcp`; if_env-skipped entries count as still spec'd); MCP orphan report is warn-only; plugin orphan report is warn-only and disabled without the plugins marker | | gh auth | `authenticate_gh` :835 | every boot when `features.gh_auth`; non-fatal | | Seed | `seed_content` :884 | workspace first boot only; skills + docs full replace every boot; `AGENT_SKIP_SEED=1` skips seeding only | | Post-startup | `post_startup` :1295 | forked child: gateway wait ≤180s, cron seed in-process, memory reindex (3 tries / 10s backoff; degraded success is retryable), STABLE doctor skills reconcile (`disable_unavailable_skills` — the check only sees enabled skills, so disables need the finding in two settle-spaced doctor runs and heals are proven by re-enable + re-check; failed heals re-disable in-boot and defer retry until the image changes via `{data}/doctor-heal-attempts`; writes batched via `--batch-json`; steady-state boots write nothing and run doctor once; `{data}/doctor-disabled-skills` marker preserves operator intent), security audit + report persistence (`{data}/logs/{doctor,security}-report.json` on findings with up-to-ten per-finding `checkId + path` log lines; warn-only, never gates), boot summary `{data}/status.json` (image version + warning count — text never reaches disk) | diff --git a/container/entrypoint.py b/container/entrypoint.py index f96f6af..bef1d11 100644 --- a/container/entrypoint.py +++ b/container/entrypoint.py @@ -254,6 +254,32 @@ def config_set(key: str, value: str, *extra: str) -> None: warn(f"config set failed: {key}") +def config_merge_list(key: str, desired_items: list) -> None: + """Union-apply a merge-flagged list entry: skip the CLI when every spec + item is already stored, otherwise write the union — current order + first, spec additions in spec order — so items added outside the spec + (doctor allowlist expansions, operator edits) survive reconcile. A + missing or non-list stored value converges to the spec list. An + unreadable config skips the drift check (mirror of config_set). + Failure warns (naming the key only) and never raises.""" + merged: list | None = None + config = read_openclaw_config() + if config is not None: + current = lookup_config_path(config, key) + if isinstance(current, list): + missing = [item for item in desired_items if item not in current] + if not missing: + config_reconcile_stats["skipped"] += 1 + return + merged = [*current, *missing] + if merged is None: + merged = list(desired_items) + config_reconcile_stats["applied"] += 1 + result = run("openclaw", "config", "set", key, json.dumps(merged), "--strict-json", check=False) + if result is not None and result.returncode != 0: + warn(f"config set failed: {key}") + + def config_set_batch(ops: list[tuple[str, object]], *, force: bool = False) -> None: """Set several config keys in ONE ``openclaw config set --batch-json`` call. Ops whose stored value already matches are dropped first (the @@ -474,7 +500,10 @@ def reconcile_config(spec: Spec, env: Mapping[str, str]) -> None: if not entry.env_guard_satisfied(env): log(f"skipped (if_env unsatisfied: {entry.path})") continue - if entry.use_strict_json: + if entry.merge: + # parser guarantees merge entries carry a strict-JSON list in cli_value + config_merge_list(entry.path, json.loads(entry.cli_value)) + elif entry.use_strict_json: config_set(entry.path, entry.cli_value, "--strict-json") else: config_set(entry.path, entry.cli_value) diff --git a/container/spec.py b/container/spec.py index 8feefbf..ba31a8c 100644 --- a/container/spec.py +++ b/container/spec.py @@ -83,6 +83,10 @@ class ConfigEntry: strict-JSON list of items. An empty result is a SpecError naming the referenced env var(s) — a configured CSV that yields nothing is a misconfiguration. + - merge=true requires the final value (post split_csv) to be a JSON list; + cli_value stays the strict-JSON list of spec items, and the entrypoint + applies it as a UNION with the stored list — items added outside the + spec (doctor allowlist expansions, operator edits) survive reconcile. """ path: str @@ -92,6 +96,7 @@ class ConfigEntry: use_strict_json: bool if_env: tuple[str, ...] = () split_csv: bool = False + merge: bool = False def env_guard_satisfied(self, env: Mapping[str, str]) -> bool: """True iff every name in if_env is present in env; empty guard always passes.""" @@ -398,7 +403,7 @@ def _expect_str_dict(value: object, path: str) -> dict[str, str]: _THINKING_LEVELS = frozenset( {"off", "minimal", "low", "medium", "high", "xhigh", "adaptive", "max"} ) -_CONFIG_ENTRY_KEYS = frozenset({"path", "value", "strict", "if_env", "split_csv"}) +_CONFIG_ENTRY_KEYS = frozenset({"path", "value", "strict", "if_env", "split_csv", "merge"}) _INCLUDE_ONLY_KEYS = frozenset({"include"}) _PRESET_NAME_RE = re.compile(r"[a-z0-9][a-z0-9_-]*") _CONFIG_KEY_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*") @@ -487,6 +492,7 @@ def _parse_config_entries( _reject_unknown_keys(node, _CONFIG_ENTRY_KEYS, base) strict = _expect_bool(node.get("strict", False), _join(base, "strict")) split_csv = _expect_bool(node.get("split_csv", False), _join(base, "split_csv")) + merge = _expect_bool(node.get("merge", False), _join(base, "merge")) if_env = tuple(_expect_str_list(node.get("if_env", []), _join(base, "if_env"))) lookup = _env_for(if_env, env) path = _resolve_string( @@ -513,9 +519,13 @@ def _parse_config_entries( _fail(_join(base, "value"), f"split_csv produced no non-empty items{hint}") cli_value = json.dumps(items) use_strict = True + final_value: JSONValue = items else: use_strict = strict or not isinstance(resolved, str) cli_value = json.dumps(resolved) if use_strict else str(resolved) + final_value = resolved + if merge and not isinstance(final_value, list): + _fail(_join(base, "merge"), "merge requires the value to resolve to a list") entries.append( ConfigEntry( path=path, @@ -525,6 +535,7 @@ def _parse_config_entries( use_strict_json=use_strict, if_env=if_env, split_csv=split_csv, + merge=merge, ) ) return entries diff --git a/container/test_entrypoint.py b/container/test_entrypoint.py index f5cc36c..e7c6295 100644 --- a/container/test_entrypoint.py +++ b/container/test_entrypoint.py @@ -410,6 +410,124 @@ def test_nested_path_through_non_dict_segment(self) -> None: self.assert_applied("agents.defaults.utilityModel", "zai/glm-5.3-flash") +class ConfigMergeMatrix(EntrypointTestCase): + """merge-flagged entries union with the stored list: items added outside + the spec (doctor allowlist expansions, operator edits) survive reconcile + instead of being reverted to the exact spec value.""" + + def call(self, key: str, items: list) -> tuple[str, str]: + return self.capture(lambda: entrypoint.config_merge_list(key, items)) + + def assert_skipped(self) -> None: + self.assertEqual([], self.calls_with("openclaw", "config", "set")) + self.assertEqual({"applied": 0, "skipped": 1}, entrypoint.config_reconcile_stats) + + def assert_applied(self, key: str, value: object) -> None: + self.assertEqual( + [["openclaw", "config", "set", key, json.dumps(value), "--strict-json"]], + self.calls_with("openclaw", "config", "set"), + ) + self.assertEqual({"applied": 1, "skipped": 0}, entrypoint.config_reconcile_stats) + + def test_missing_key_writes_spec_items(self) -> None: + self.write_openclaw_config('{"plugins": {}}') + self.call("plugins.allow", ["grow-approval-gate", "llama-cpp"]) + self.assert_applied("plugins.allow", ["grow-approval-gate", "llama-cpp"]) + + def test_missing_items_union_preserves_current_order(self) -> None: + self.write_openclaw_config('{"plugins": {"allow": ["llama-cpp", "memory-core"]}}') + self.call("plugins.allow", ["grow-approval-gate", "llama-cpp"]) + self.assert_applied("plugins.allow", ["llama-cpp", "memory-core", "grow-approval-gate"]) + + def test_superset_skips_cli(self) -> None: + self.write_openclaw_config( + '{"plugins": {"allow": ["grow-approval-gate", "llama-cpp", "memory-core"]}}' + ) + self.call("plugins.allow", ["grow-approval-gate", "llama-cpp"]) + self.assert_skipped() + + def test_exact_match_skips_cli(self) -> None: + self.write_openclaw_config('{"plugins": {"allow": ["a", "b"]}}') + self.call("plugins.allow", ["a", "b"]) + self.assert_skipped() + + def test_non_list_current_converges_to_spec(self) -> None: + self.write_openclaw_config('{"plugins": {"allow": "wildcard"}}') + self.call("plugins.allow", ["a", "b"]) + self.assert_applied("plugins.allow", ["a", "b"]) + + def test_missing_config_file_writes_spec_items(self) -> None: + self.assertFalse((self.data / "openclaw.json").exists()) + self.call("plugins.allow", ["a", "b"]) + self.assert_applied("plugins.allow", ["a", "b"]) + + def test_malformed_config_file_writes_spec_items(self) -> None: + self.write_openclaw_config('{"plugins": "not closed') + self.call("plugins.allow", ["a", "b"]) + self.assert_applied("plugins.allow", ["a", "b"]) + + def test_failed_write_warns_with_key_but_never_value(self) -> None: + def failing(cmd: list[str]) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(cmd, 1, stdout="", stderr="") + + self.handler = failing + _out, err = self.call("plugins.allow", ["secret-plugin"]) + self.assertIn("config set failed: plugins.allow", err) + + def test_reconcile_applies_merge_entry_as_union(self) -> None: + spec_doc = copy.deepcopy(MINIMAL_SPEC) + spec_doc["config"] = [ + { + "path": "plugins.allow", + "value": ["grow-approval-gate", "llama-cpp"], + "merge": True, + } + ] + spec = self.load_spec_with(spec_doc) + self.write_openclaw_config('{"plugins": {"allow": ["llama-cpp", "memory-core"]}}') + self.capture(lambda: entrypoint.reconcile_config(spec, os.environ)) + self.assertEqual( + [ + [ + "openclaw", + "config", + "set", + "plugins.allow", + json.dumps(["llama-cpp", "memory-core", "grow-approval-gate"]), + "--strict-json", + ], + [ + "openclaw", + "config", + "set", + "tools.deny", + json.dumps(list(entrypoint.TOOLS_DENY_DEFAULT)), + "--strict-json", + ], + ["openclaw", "config", "set", "gateway.bind", "lan"], + ], + self.calls_with("openclaw", "config", "set"), + ) + + def test_reconcile_skips_merge_entry_when_guard_unsatisfied(self) -> None: + spec_doc = copy.deepcopy(MINIMAL_SPEC) + spec_doc["config"] = [ + { + "path": "plugins.allow", + "value": ["a"], + "merge": True, + "if_env": ["TELEGRAM_CHAT_ID"], + } + ] + spec = self.load_spec_with(spec_doc) + out, _err = self.capture(lambda: entrypoint.reconcile_config(spec, os.environ)) + self.assertIn("skipped (if_env unsatisfied: plugins.allow)", out) + allow_calls = [ + c for c in self.calls_with("openclaw", "config", "set") if c[3] == "plugins.allow" + ] + self.assertEqual([], allow_calls) + + class FirstBootSequence(EntrypointTestCase): SETUP_CMD = [ "openclaw", diff --git a/container/test_spec.py b/container/test_spec.py index 5abed44..67114fd 100644 --- a/container/test_spec.py +++ b/container/test_spec.py @@ -422,6 +422,49 @@ def test_malformed_values_fail_closed(self) -> None: ) +class MergeConfigEntries(SpecTestCase): + """config entry `merge` flag (default false): union semantics for list + values — cli_value stays the strict-JSON spec list and the entrypoint + unions it with the stored list instead of overwriting it.""" + + def test_merge_defaults_false(self) -> None: + loaded = self.load(self.config_spec(["a"]), env={"ZAI_API_KEY": "zai-key"}) + self.assertFalse(loaded.config_entries[0].merge) + + def test_merge_list_value_parses(self) -> None: + loaded = self.load( + self.config_spec(["grow-approval-gate", "llama-cpp"], merge=True), + env={"ZAI_API_KEY": "zai-key"}, + ) + entry = loaded.config_entries[0] + self.assertTrue(entry.merge) + self.assertTrue(entry.use_strict_json) + self.assertEqual('["grow-approval-gate", "llama-cpp"]', entry.cli_value) + + def test_merge_with_split_csv_parses(self) -> None: + loaded = self.load( + self.config_spec("grow-approval-gate, llama-cpp", merge=True, split_csv=True), + env={"ZAI_API_KEY": "zai-key"}, + ) + entry = loaded.config_entries[0] + self.assertTrue(entry.merge) + self.assertEqual('["grow-approval-gate", "llama-cpp"]', entry.cli_value) + + def test_merge_scalar_value_fails_closed(self) -> None: + self.load_expect_error( + self.config_spec("allowlist", merge=True), + env={"ZAI_API_KEY": "zai-key"}, + containing="merge", + ) + + def test_merge_object_value_fails_closed(self) -> None: + self.load_expect_error( + self.config_spec({"deep": {"merge": True}}, merge=True), + env={"ZAI_API_KEY": "zai-key"}, + containing="merge", + ) + + class PluginPruneFeature(SpecTestCase): """features.plugin_prune (default false): opt-in removal of de-specified plugins the base itself installed (ownership via diff --git a/docs/standard-agent.md b/docs/standard-agent.md index a7ee322..b7b193a 100644 --- a/docs/standard-agent.md +++ b/docs/standard-agent.md @@ -319,7 +319,7 @@ and every error message starts with the JSON path of the offending node | `setup` | `auth_choice` | Required. Passed to `openclaw setup --auth-choice` on first boot (e.g. `litellm-api-key`, `zai-coding-global`). `zai-coding-*` choices require `ZAI_API_KEY`; `litellm-api-key` requires `LITELLM_API_KEY` (see [Environment contract](#environment-contract)). | | `model` | `fallback`, `thinking` | `fallback` required; registered via `openclaw models fallbacks add` on first boot. `thinking` optional: the reasoning effort, seeded to `agents.defaults.thinkingDefault` (OpenClaw's global thinking default, resolved per turn) on every boot unless a config entry owns the path. Allowed values: `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `adaptive`, `max`; anything else is a load error. Providers map the level onto their own request field (e.g. the Z.AI provider sends `reasoning_effort`; for GLM-5.x an explicit `off` still maps to that API's lowest effort). The base merges `fallback` into an existing `agents.defaults.models` allowlist (see [Boot sequence](#boot-sequence)). | | `automations` | `model`, `default_tools` | `model` required: the model for cron agent turns. No default exists by design (see [Standardization decisions](#standardization-decisions)). `default_tools` optional: the default tool allow-list for seeded jobs without their own `tools:` header; `["*"]` restores unrestricted. | -| `config` | `path`, `value`, `strict`, `if_env`, `split_csv` | `path` and `value` required; the rest are optional booleans / string lists. Applied in spec order. `path` accepts `{env:...}` tokens (chat IDs stop being baked into git); resolution mirrors `value`, including guard deferral. An item may instead be exactly `{"include": ""}.` See [config entries](#config-entries). | +| `config` | `path`, `value`, `strict`, `if_env`, `split_csv`, `merge` | `path` and `value` required; the rest are optional booleans / string lists. Applied in spec order. `path` accepts `{env:...}` tokens (chat IDs stop being baked into git); resolution mirrors `value`, including guard deferral. An item may instead be exactly `{"include": ""}.` See [config entries](#config-entries). | | `presets` | name → list of config entries | Named groups spliced into `config` in place via `{"include": ...}`. Names are `lowercase-identifier-ish`. Fail-closed: unknown names, nesting (include inside a preset), and malformed shapes are load errors. Spliced entries behave identically to inline ones (templating, guard deferral, split_csv). | | `channels` | `type`, `use_env` | `type` required. `use_env` (default `true`) feeds the channel credentials from the environment; the pinned CLI (2026.10+) validates them at add time, so a missing credential aborts first boot with the env var named by the CLI. | | `mcp_servers` | `name`, `command` or `url`, `args`, `env`, `headers`, `no_probe`, `timeout`, `if_env`, `config`, `auth`, `oauth`, `transport` | Exactly one of `command` (local stdio) and `url` (remote HTTP); specifying both or neither is an error. See [mcp_servers entries](#mcp_servers-entries). | @@ -365,6 +365,13 @@ from the resolved value: `TELEGRAM_ALLOWED_USERS=123,456` becomes a real array config value. An empty result is a load error naming the referenced env var(s); a configured CSV that yields nothing is a misconfiguration, not a skip. +- `merge: true`: the value (post `split_csv`) must resolve to a JSON list. + Instead of overwriting the stored list, reconcile unions it with the + spec's items — current order first, additions in spec order — and skips + the CLI entirely when every spec item is already present. This is how a + spec-owned `plugins.allow` coexists with plugins that `openclaw doctor` + or the operator adds later: the spec guarantees its own items are + present, it never reverts anyone else's. - `if_env`: the entry is applied only when every listed variable is present. An unsatisfied guard is a logged skip, never an error. Guards must pair the knobs they make optional: `channels.telegram.dmPolicy` @@ -377,7 +384,9 @@ from the resolved value: On warm volumes the entrypoint compares each desired value against `openclaw.json` directly and shells out only for keys that actually drift, -so a rebooting container costs one JSON read, not N CLI spawns. +so a rebooting container costs one JSON read, not N CLI spawns. Merge +entries skip on subset containment rather than exact equality, so items +added to the stored list outside the spec never look like drift. ### mcp_servers entries