Add isolated test-environment host installer - #76
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b86cb94df9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review Please confirm exact head a75e2fa69afe6bff6b4f6a2b3f211b80495bd65b after the accepted lifecycle, policy, rollback, health, and release-integrity fixes. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a75e2fac15
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review Please confirm exact head f36fa02c3188de680087913e34bcb4f739641bd6; this only makes read-only release tamper/teardown tests work under non-root CI, with the full tester suite passing. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f36fa02c6d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 196b121df6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca39b25234
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4736e5e1a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 841406c946
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d88df91e1d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3cc48dc63
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08be87de50
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| key=r'(?:!!str[ \t]+)?(?:include|label_file|"(?:include|label_file)"|\x27(?:include|label_file)\x27)[ \t]*:' | ||
| explicit_key=r'(?m)^[ \t]*\?' | ||
| escaped_key=r'"[^"\n]*\\[^"\n]*"[ \t]*:' | ||
| if re.search(r'(?m)^[ \t]*'+key,text) or re.search(r'[,{][ \t]*'+key,text) or re.search(escaped_key,text) or re.search(explicit_key,text): raise SystemExit('Compose include, label_file, or escaped mapping key is forbidden') |
There was a problem hiding this comment.
When a definition uses services.*.extends.file, Compose loads and merges another Compose file before this rendered-model policy runs; the Compose extends reference defines file as the location of another Compose configuration file. Because the source scan only rejects include and label_file, an extended file can contain label_file pointing at another environment's host credential, which Compose resolves into ordinary labels and leaves no forbidden field for the later predicate to detect. Reject extends before rendering or recursively validate every referenced source within the same protected boundary.
AGENTS.md reference: AGENTS.md:L33-L34
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9654ace274
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 55db13afac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8a517afeb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1892638f52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 818b78024f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8087c375ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1835871f2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| docker info --format '{{.DockerRootDir}}' >/dev/null | ||
| docker compose version >/dev/null | ||
| getent ahosts "$probe_host" >/dev/null || die 'test-host DNS probe failed' | ||
| curl --fail --silent --show-error --head --max-time 10 --output /dev/null "$probe_url" || die 'test-host HTTPS/proxy probe failed' |
There was a problem hiding this comment.
During installation or a manual --check, an inherited HTTPS_PROXY or ALL_PROXY makes this request traverse that proxy instead of the intended local test-host route, so a reachable external proxy can produce a successful check while the configured HTTPS/local-proxy path is broken. Curl documents --noproxy as the option controlling which hosts bypass proxies (curl manual); clear proxy variables or use --noproxy '*' for this probe.
AGENTS.md reference: AGENTS.md:L34-L34
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54287831b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 589fc4e284
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| done | ||
| [[ -n $action ]] || { usage; exit 2; } | ||
| case $action in --converge|--reset|--remove|--inspect) [[ $environment =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] || die 'environment ID is invalid' ;; *) [[ -z $environment ]] || die '--environment is not valid for this action' ;; esac | ||
| for command in awk basename chmod cmp curl date df dirname docker du env find flock getent grep install mktemp mv python3 readlink rm stat wc; do command -v "$command" >/dev/null || die "required command is unavailable: $command"; done |
There was a problem hiding this comment.
When curl or getent is removed or temporarily unavailable, the scheduled --cleanup service exits here before processing expirations even though that action never uses either command. This leaves expired environments running until the unrelated network-probe tooling is restored; select prerequisites by action so cleanup retains only the tools it actually needs.
AGENTS.md reference: AGENTS.md:L55-L57
Useful? React with 👍 / 👎.
| item={'source':secret,'target':secret} if isinstance(secret,str) else secret | ||
| expected_mounts.append(('bind',model['secrets'][item['source']]['file'],f"/run/secrets/{item.get('target',item['source'])}",False)) | ||
| expected_mounts=sorted(expected_mounts) | ||
| actual_mounts=sorted((mount.get('Type'),mount.get('Source','') if mount.get('Type')=='bind' else '',mount.get('Destination'),bool(mount.get('RW'))) for mount in actual.get('Mounts') or []) |
There was a problem hiding this comment.
Fresh evidence after the live-isolation fix is that this tuple deliberately records an empty source for every non-bind mount. If a project container is recreated externally with the expected labels, image, limits, and target path but mounts another environment's named volume, both the expected and actual mounts reduce to ('volume', '', target, rw), so --health reports it as valid while cross-environment data is exposed or modified; compare the inspected volume name with the resolved Compose volume name.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57db054a6d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| escaped_key=r'"[^"\n]*\\[^"\n]*"[ \t]*:' | ||
| alias_key=r'(?m)(?:^[ \t]*|[,{][ \t]*)\*[A-Za-z0-9_-]+[ \t]*:' | ||
| anchor_key=r'(?m)(?:^[ \t]*|[,{][ \t]*)&[A-Za-z0-9_-]+[ \t]+' | ||
| if re.search(r'(?m)^[ \t]*'+key,text) or re.search(r'[,{][ \t]*'+key,text) or re.search(escaped_key,text) or re.search(explicit_key,text) or re.search(alias_key,text) or re.search(anchor_key,text): raise SystemExit('Compose include, label_file, or indirect mapping key is forbidden') |
There was a problem hiding this comment.
On Compose versions supporting include, a valid YAML key such as !!str &external include: [{path: /root/other.yaml}] bypasses every expression here: key allows a tag only immediately before the key, while anchor_key requires the anchor to be the first node property. Compose decodes this as the ordinary include key and can load an external definition or its interpolation file before rendered-model validation, potentially persisting host credential values in allowed labels or commands. Fresh evidence beyond the prior anchored-key fix is the valid tag-first tag-and-anchor combination; reject decoded source keys rather than enumerating YAML syntax permutations.
AGENTS.md reference: AGENTS.md:L33-L34
Useful? React with 👍 / 👎.
| valid=(host.get('Privileged') is False and host.get('ReadonlyRootfs') is True and set(host.get('CapDrop') or [])=={'ALL'} and | ||
| not host.get('CapAdd') and security(host.get('SecurityOpt'))==security(service['security_opt']) and | ||
| host.get('NanoCpus')==int(service['cpus']*1_000_000_000) and host.get('Memory')==service['mem_limit'] and | ||
| host.get('PidsLimit')==service['pids_limit'] and (host.get('PortBindings') or {})==expected_ports and actual_mounts==expected_mounts) |
There was a problem hiding this comment.
Fresh evidence after the prior live-isolation fix is that a container recreated with the expected labels, image, limits, mounts, and port-binding metadata still passes this predicate when HostConfig.NetworkMode is host, or when it has been connected to an additional Docker network, because neither the namespace mode nor NetworkSettings.Networks is compared with the protected model. In that scenario --health reports the environment as valid despite access beyond its approved Compose networks; compare the resolved network mode and complete attachment set before accepting the container.
AGENTS.md reference: AGENTS.md:L33-L34
Useful? React with 👍 / 👎.
| docker compose -p "$(project_name "$id")" -f "$compose" down --timeout 10 --volumes --remove-orphans || return 1 | ||
| rm -f -- "$target" "$(deployed_compose_path "$id")" | ||
| fi | ||
| report "REMOVED environment=$id" |
There was a problem hiding this comment.
When an environment's .state file is lost while its Compose project is still running, --remove --environment <id> skips teardown and nevertheless prints REMOVED. Fresh evidence after orphan checks were added to scheduled cleanup and uninstall is that this explicit removal path calls neither check, so the operator receives a false success while containers and volumes remain and subsequent cleanup can only report an unrecoverable orphan; inspect the project inventory and fail rather than reporting removal when state is absent.
AGENTS.md reference: AGENTS.md:L66-L66
Useful? React with 👍 / 👎.
| cmp -s "$release_dir/$current/scripts/tester-launcher.sh" "$stable_launcher" || die 'installed launcher differs from active release' | ||
| for unit in "${units[@]}"; do | ||
| secure_file "$systemd_dir/$unit" 644 | ||
| cmp -s "$release_dir/$current/host/systemd/$unit" "$systemd_dir/$unit" || die "installed unit differs from active release: $unit" |
There was a problem hiding this comment.
When the host already contains /etc/systemd/system/ci-fleet-tester-health.service.d/*.conf or an equivalent cleanup/timer drop-in, systemd merges it after the validated main fragment as described by the systemd unit drop-in semantics, but installation and --check inspect only the main file. A stale override can replace ExecStart, clear sandboxing, or redirect a timer, allowing activation and later checks to succeed while the effective root unit differs from the reviewed release. Fresh evidence beyond the prior main-unit ownership and content validation is that it does not cover merged drop-ins; reject unmanaged drop-ins or validate the complete effective unit policy.
AGENTS.md reference: AGENTS.md:L34-L34
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: de02123854
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if ! python3 - "$compose_file" <<'PY' | ||
| import re,sys | ||
| text=open(sys.argv[1],encoding='utf-8').read() | ||
| key=r'(?:(?:!!str|!<[^>\n]+>)[ \t]+)?(?:include|label_file|"(?:include|label_file)"|\x27(?:include|label_file)\x27)[ \t]*:' |
There was a problem hiding this comment.
Fresh evidence beyond the include/label_file scanner is that Docker Compose also allows extends.file to load a service definition from another Compose file. Because extends is absent from this pre-render check, a reviewed definition can point outside the protected definitions directory; Compose resolves that external content before the rendered-model validator sees it, allowing otherwise-permitted fields such as labels or commands to embed host credential values. Reject file-backed extends before invoking Compose.
AGENTS.md reference: AGENTS.md:L33-L34
Useful? React with 👍 / 👎.
| inventory=$(docker volume ls -q --filter "label=com.docker.compose.project=$(project_name "$id")") || die 'volume inventory failed' | ||
| while IFS= read -r resource; do | ||
| [[ -n $resource ]] || continue | ||
| mount=$(docker volume inspect --format '{{.Mountpoint}}' "$resource"); value=$(du -sb "$mount" | awk '{print $1}'); [[ $value =~ ^[0-9]+$ ]] || die 'volume disk size is invalid'; bytes=$((bytes + value)) |
There was a problem hiding this comment.
Fresh evidence beyond the pre-convergence resource check is that scheduled health only inventories project-labeled volumes and measures their mountpoints; it never verifies their live driver or options. If an active environment's named volume is replaced with an identically named and labeled local-driver bind volume and its container is recreated, the container check still sees an expected volume mount while the workload can access the bound host path, and --health reports success. Apply the existing driver/options/provenance validation to live volumes before accepting the environment.
AGENTS.md reference: AGENTS.md:L33-L34
Useful? React with 👍 / 👎.
| valid=(host.get('Privileged') is False and host.get('ReadonlyRootfs') is True and set(host.get('CapDrop') or [])=={'ALL'} and | ||
| not host.get('CapAdd') and security(host.get('SecurityOpt'))==security(service['security_opt']) and | ||
| host.get('NanoCpus')==int(service['cpus']*1_000_000_000) and host.get('Memory')==service['mem_limit'] and | ||
| host.get('PidsLimit')==service['pids_limit'] and (host.get('PortBindings') or {})==expected_ports and actual_mounts==expected_mounts) |
There was a problem hiding this comment.
Fresh evidence beyond the rendered-model OOM check is that this live predicate omits HostConfig.OomKillDisable and OomScoreAdj. A container recreated with the expected image, labels, limits, mounts, and ports but with OOM killing disabled or a negative OOM score therefore passes scheduled health despite violating the policy enforced at convergence and potentially impairing the shared daemon under memory pressure; compare both live values with the approved defaults.
AGENTS.md reference: AGENTS.md:L33-L33
Useful? React with 👍 / 👎.
| if [[ $service == "$route_service" ]]; then | ||
| [[ $running_state == 'running healthy' ]] || status=unhealthy |
There was a problem hiding this comment.
When the route service has a Docker health check that succeeds without listening on the declared application port—for example, a check that merely runs true—this branch accepts running healthy, while the live-policy check verifies only port-binding metadata rather than reachability. Consequently convergence and the scheduled five-minute health check can report success even though nothing is reachable through the advertised loopback/local-proxy route; perform a bounded connection to the declared route before returning STATUS=running.
Useful? React with 👍 / 👎.
1 participant
Closes #23
Summary
Validation
scripts/test-install-tester.sh(TESTER_INSTALLER_TESTS_OK)scripts/validate.shreachedINSTALLER_TESTS_OKandSTATUS_RECEIVER_INSTALL_TESTS_OK; final Docker smoke was omitted because this repository-only environment has no Docker CLI/daemon (docker: command not found)External gate
Draft: issue acceptance still requires install/upgrade/rollback/cleanup evidence on a prepared Debian test host. This run did not touch a live host, Docker daemon, systemd, network, secret, test environment, or production system.