diff --git a/docs/PRD.md b/docs/PRD.md index 8ce7f507..5a86d13a 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -51,9 +51,9 @@ The runtime is deliberately not the authority for the consumer's business decisi - Enforce CPU, RAM, PID, tmpfs, lease, readiness, and shutdown bounds. - Create a per-sandbox internal DNS-disabled network and publish exactly one service to host IPv4 loopback on a random port. - Invoke the application without a shell. -- Return an endpoint only after bounded readiness. +- Return an endpoint only after protocol-aware bounded readiness: TCP services require loopback transport reachability; declared HTTP services require a successful bounded HTTP response from the runtime-derived loopback endpoint. - Return a versioned lease and cleanup receipt. -- Provide no request fields for credentials, arbitrary environment variables, broad host mounts, devices, privileged mode, host namespaces, runtime sockets, or arbitrary Internet egress. +- Provide no request fields for credentials, arbitrary environment variables, broad host mounts, devices, privileged mode, host namespaces, runtime sockets, arbitrary health URLs, or arbitrary Internet egress. ### Artifact-analysis foundation @@ -110,7 +110,7 @@ The runtime is deliberately not the authority for the consumer's business decisi - Existing artifact-analysis public Rust API remains available after DDD directory migration. - Tag-only images and over-budget resource requests fail closed. - Launch plan has no privileged/host-network/runtime-socket path and explicitly enforces P0 isolation flags. -- Process-boundary tests prove direct argv invocation, readiness gating, error cleanup, and explicit termination behavior. +- Process-boundary tests prove direct argv invocation, protocol-aware readiness gating, error cleanup, and explicit termination behavior; an HTTP declaration cannot become ready from TCP acceptance alone, and a non-2xx HTTP response is not readiness. - Real rootless Podman E2E proves the effective security boundary before the capability is called release-ready. - `contextual-orchestrator` integration occurs through its owner issue/ACL and a published runtime artifact; no direct consumer Podman calls. - Wardnet verdict policy remains outside this repository. diff --git a/docs/TRD.md b/docs/TRD.md index 7c186724..f0c7f62b 100644 --- a/docs/TRD.md +++ b/docs/TRD.md @@ -147,10 +147,10 @@ The immutable image reference is appended before application argv, so applicatio 5. Inspect process seccomp/capability/LSM evidence and fail closed unless every implemented P0 isolation control is positively verified. 6. Query the requested port mapping only after isolation verification succeeds. 7. Accept only a single IPv4 loopback `127.0.0.1:` mapping. -8. Poll bounded TCP readiness using operator timeout/poll policy. +8. Poll protocol-aware readiness within the operator-owned timeout/poll budget: `tcp` requires a successful loopback TCP connection; `http` sends a bounded HTTP/1.1 request to the runtime-derived loopback endpoint and requires a final 2xx status-class response before readiness is accepted. 9. Return a lease only after effective-isolation checks and readiness succeed. -P0 HTTP readiness deliberately uses TCP reachability because no consumer-supplied health path is accepted yet. A future typed HTTP health contract may refine this without accepting arbitrary URLs. +P0 HTTP readiness never accepts a caller-supplied URL, origin, host, or arbitrary health path. The probe is fixed to `/` on the runtime-owned loopback mapping, so consumer-specific authentication or bootstrap semantics remain outside this bounded context. A future typed application-specific health contract may version this boundary without turning arbitrary network destinations into runtime authority. ### Cleanup diff --git a/docs/adr/0006-isolated-application-service.md b/docs/adr/0006-isolated-application-service.md index 023e324c..70ab8345 100644 --- a/docs/adr/0006-isolated-application-service.md +++ b/docs/adr/0006-isolated-application-service.md @@ -9,6 +9,8 @@ This ADR remains Proposed while PR #1 is Draft and the exact candidate has not p Chat/Agent systems sometimes need a real application process rather than an in-process tool function. Running such applications on the orchestrator host with ambient credentials, host networking, writable filesystem, or container-engine access creates an unacceptable authority boundary. +A loopback port accepting TCP is not sufficient evidence that an application declared as HTTP is ready for a consumer security bootstrap. The runtime therefore needs protocol-aware readiness while keeping consumer-specific login, authorization, and arbitrary health URLs outside the isolation bounded context. + ## Decision The first `application_service` infrastructure adapter is rootless Podman. A valid request must use an immutable OCI image digest and bounded service/resources. The adapter: @@ -24,19 +26,23 @@ The first `application_service` infrastructure adapter is rootless Podman. A val - disables restart and container logging in the P0 profile; - applies CPU/RAM/PID and container lifetime limits; - publishes one service port to random host port on `127.0.0.1` only; -- performs a bounded readiness check before returning a lease; +- performs protocol-aware bounded readiness before returning a lease: `tcp` requires a successful loopback connection, while `http` sends a fixed HTTP/1.1 request to `/` on the runtime-derived loopback endpoint and requires a final 2xx status-class response; +- never accepts a caller-supplied readiness URL, origin, host, or arbitrary path in the P0 contract; - removes container and network on explicit termination and on partial-launch/readiness failures; - returns an attested versioned lease and cleanup receipt. -The application request has no fields for privileged mode, host namespaces, devices, arbitrary mounts, environment variables, runtime sockets, or external network enablement. +The application request has no fields for privileged mode, host namespaces, devices, arbitrary mounts, environment variables, runtime sockets, external network enablement, or caller-controlled health destinations. Consumer-specific authentication/bootstrap remains consumer-owned and occurs only after the runtime has returned an attested ready lease. ## Alternatives - **Consumer-owned Podman calls:** rejected because isolation policy would be duplicated and domain code would depend on infrastructure. +- **TCP reachability for HTTP readiness:** rejected because a listening socket can exist before the declared HTTP service is usable and can issue a lease that immediately fails consumer bootstrap. +- **Caller-supplied health URL/path:** rejected for P0 because it expands the runtime into an outbound-request authority and mixes consumer application semantics into the isolation contract. +- **Consumer login as runtime readiness:** rejected because login credentials and application authorization belong to the consumer, not the reusable isolation runtime. - **Docker socket sidecar:** rejected because the socket is a high-authority control channel. - **gVisor first:** deferred. gVisor is a planned stronger OCI backend but P0 first establishes the consumer-neutral contract and rootless lifecycle with broadly available Podman. - **Kubernetes first:** deferred because a cluster is not required for standalone/local operation. ## Verification rule -Command-plan and fake-Podman process tests are necessary but not sufficient. Release claims about container isolation require a real rootless Podman E2E lane that checks effective filesystem, namespace, capability, resource, network, and cleanup behavior. gVisor/containerd/Kubernetes adapters require their own parity and security evidence. +Command-plan and fake-Podman process tests are necessary but not sufficient. Protocol readiness tests must prove that TCP-only acceptance does not satisfy an HTTP declaration, that a bounded 2xx response does, and that non-2xx/stalled responses fail closed. Release claims about container isolation require a real rootless Podman E2E lane that checks effective filesystem, namespace, capability, resource, network, and cleanup behavior. gVisor/containerd/Kubernetes adapters require their own parity and security evidence. diff --git a/docs/doctoring/APPLICATION_SERVICE_HTTP_READINESS_GAP_OWNER_REPAIR.md b/docs/doctoring/APPLICATION_SERVICE_HTTP_READINESS_GAP_OWNER_REPAIR.md new file mode 100644 index 00000000..2c036a41 --- /dev/null +++ b/docs/doctoring/APPLICATION_SERVICE_HTTP_READINESS_GAP_OWNER_REPAIR.md @@ -0,0 +1,35 @@ +# Application-service HTTP-readiness Gap owner repair + +Reviewed 2026-09-17 KST against Draft PR #104 exact `4d738ccc52a3acb3d6ea621e306f628074c96122`, exact base `5c6a44bb2b35eb17d0315d72db242f4488c3c426`, and repository-wide Gap owner PR #121. + +## Owner-boundary finding + +Issue #103 is a focused application-service readiness contract. It owns protocol-aware readiness semantics, the Podman readiness adapter, its focused tests, and the corresponding PRD/TRD/ADR/TRACEABILITY updates. It does not own the repository-wide live product/technical Gap ledger. Review `5229919282` found that the branch still changed `docs/product-technical-gap-baseline.md`; carrying that file forward could replay a 2026-09-09 repository snapshot over #121's newer owner graph. + +The repair is migration-first. The protocol evidence and consumer-release rule that had also appeared in the global ledger are retained here and in the existing local PRD/TRD/ADR/TRACEABILITY before the global file is restored byte-for-byte to this PR's exact-base blob `bacb346f2ce4259a4f55bd3bece5e871b06d69db`. + +## Retained causal and review evidence + +The repaired causal RED `65663052ec30bc178adbe5ff4514f5409d10971f`, native CI `34255729573`, verify `102160927453`, proved that a runtime-owned loopback socket accepting TCP without an HTTP response could incorrectly satisfy `ServiceProtocol::Http`. The first production repair `c05d378cfc736e4257594d69bb06871893ef2d0f` split protocol semantics: `Tcp` remains connect-only; `Http` sends one bounded fixed-path request to the runtime-owned loopback mapping and requires a successful HTTP response class. + +Legacy success fixtures that were plain TCP listeners were corrected to `Tcp` in `59cc738f1428d78eaf7a7999e65cc247307b990d`. A focused helper-coverage repair followed without widening readiness semantics. Exact predecessor `d098385045cefd8b337ba2bd0069107a01756b81`, CI `34259463376`, reached hosted GREEN for verify, coverage, branch coverage, and hosted negative rootless/AppArmor; its branch artifact and coverage details remain historical predecessor evidence only. + +Code review then identified two narrower protocol-integrity defects: the fixed `Host: 127.0.0.1` omitted the runtime-selected non-default port, and status acceptance treated a prefix beginning with `2` as sufficient. Test-only `32972162bfee95112b2f79a3427ebe4815e24b39`, CI `34267331198`, branch job `102199888859`, executed the causal review RED: the captured request omitted `:`, while malformed `HTTP/1.1 2x0 ...` and `HTTP/1.1 20x ...` were incorrectly accepted. The same focused binary retained TCP/no-response timeout, HTTP 204 success, and HTTP 503 non-readiness controls. + +Minimum production `b481086cbd13a1e94cf8df09d49efb9fa3200e85` derives `Host: 127.0.0.1:` and validates a bounded HTTP/1.1 three-digit 2xx status prefix. It does not add caller-controlled origin/path, authentication, credential handling, egress policy, model/provider semantics, or new wire fields. RFC 9110/9112 basis and the exact contract distinction remain in `APPLICATION_SERVICE_HTTP_READINESS_TRACEABILITY.md`. + +Exact predecessor `4d738ccc52a3acb3d6ea621e306f628074c96122`, native CI `34270054863`, had verify `102209085115`, production coverage `102209084826`, branch coverage `102209084792`, and hosted negative rootless/AppArmor `102209084617` GREEN. Dedicated positive-LSM `102209085130` was still queued and no qualifying approval existed. This hosted GREEN must not transfer after the ownership-only head movement. + +## Consumer and release contract retained locally + +The PR's PRD/TRD/ADR changes make the ownership boundary normative: consumers must consume an immutable released runtime contract/artifact; mutable PR heads, sibling source copies, direct foreign runtime calls, and cross-service SQL are not integration mechanisms. No GitHub Release is claimed by this branch. Protected integration and immutable version/package/SBOM/provenance/reproducibility/rollback publication remain prerequisites before a consumer version/digest bump. + +Caido login/authentication and consumer bootstrap remain consumer-owned. The runtime readiness contract only attests the selected backend-neutral service protocol against the exact runtime-owned loopback mapping. + +## DDD and decision + +`application_service` owns protocol intent and lease/readiness semantics. `infrastructure::podman` owns the concrete loopback probe translation. `sandbox_execution` remains the reusable isolation owner. Requested protocol intent, successful readiness observation, and positive sandbox confinement are independent evidence dimensions. + +Selected repair: preserve all focused source/test/PRD/TRD/ADR/TRACEABILITY changes, add this owner-repair record, and restore only the global Gap file to the exact base. Rejected alternatives are copying #121's latest ledger into this leaf, discarding review/causal history, retaining a second live Gap writer, or force-rebasing merely to remove the file. + +After the docs-only movement, all current-head gates must be reacquired. Historical exact-head GREEN remains useful lineage evidence but is not merge/release authority for the moved head. diff --git a/docs/doctoring/APPLICATION_SERVICE_HTTP_READINESS_TRACEABILITY.md b/docs/doctoring/APPLICATION_SERVICE_HTTP_READINESS_TRACEABILITY.md new file mode 100644 index 00000000..44428728 --- /dev/null +++ b/docs/doctoring/APPLICATION_SERVICE_HTTP_READINESS_TRACEABILITY.md @@ -0,0 +1,81 @@ +# Application-service HTTP readiness traceability + +## Status and owner + +Proposed on 2026-09-09. Issue #103 / Draft PR #104 owns the first HTTP-readiness repair inside the existing `application_service` Supporting bounded context and `sandbox_execution` Core lifecycle. It does not create a separate readiness product, move Strix/Caido application authorization into this repository, or assign LLM/provider failure taxonomy to this runtime. + +Canonical dependency root for this lane is Draft PR #1 exact `5c6a44bb2b35eb17d0315d72db242f4488c3c426`; protected/default `develop` is `60a85c7633e03b425b67159ec6822c8178cf87ea`. + +## Production evidence that motivated the RED + +The consumer failure recorded in issue #103 occurred before a Strix model-backed scan when the local Caido endpoint on `127.0.0.1:48080` never became usable. The centralized caller correctly failed closed as `STRIX_SANDBOX_UNAVAILABLE`; adding retries at the consumer would not prove sandbox readiness. + +On the canonical root, `RootlessPodmanAdapter::launch_at` verified isolation and the loopback mapping, then called `wait_for_readiness`. Before this lane, that function treated a successful TCP connection as sufficient readiness for both `ServiceProtocol::Tcp` and `ServiceProtocol::Http`. + +The buyer-visible gap was therefore narrower than the original standalone #104 prototype: an HTTP application could accept TCP while its HTTP surface was still unavailable, yet the runtime could issue a ready lease. + +## Rejected predecessor design + +Exact predecessor #104 `cd6316dc7172e97cce2630e698aa947e6df15640` created a second root Cargo package and repository CI directly from protected `develop`. Native CI `34254527569` failed at `cargo test --all-targets` before Clippy or formatting; the branch also declared a missing `src/main.rs` and its test source contained malformed byte-string/JSON literals. That is a repository/test-harness failure, not causal readiness evidence. + +The predecessor also exposed caller-controlled arbitrary `origin` and reserved `provider_terminated` / `model_communication_failed` inside the runtime. Those authorities are rejected. The runtime owns runtime-generated loopback service readiness and cleanup. Provider/model communication classification remains with the LLM/orchestration owner. + +## Causal RED and minimum behavior repair + +Exact `65663052ec30bc178adbe5ff4514f5409d10971f` established the intended RED: a listening TCP socket that deliberately produced no HTTP response was nevertheless accepted as `ServiceProtocol::Http` readiness. The expected contract was `ReadinessTimeout` plus cleanup. + +Minimum production candidate `c05d378cfc736e4257594d69bb06871893ef2d0f` changed only readiness semantics: + +- `tcp` remains successful loopback transport connection; +- `http` sends a fixed HTTP/1.1 request to `/` on the runtime-derived loopback mapping; +- a final 2xx status class is required before readiness succeeds; +- read/write I/O is bounded by the remaining operator readiness budget; +- no caller-controlled URL, origin, host, path, credential, login, or provider/model classification enters the runtime contract. + +Focused regressions prove TCP acceptance alone is not HTTP readiness, HTTP 204 is accepted, and HTTP 503 is not readiness. + +## Broad-fixture and coverage RCA + +The first exact candidate CI `34257036516` exposed an inherited fixture mismatch: `tests/podman_application_service.rs` used `ServiceProtocol::Http` while its success fixtures were intentionally plain TCP listeners. Commit `59cc738f1428d78eaf7a7999e65cc247307b990d` preserved those process-boundary fixtures as TCP instead of weakening HTTP readiness. Exact CI `34258071256` then made verify GREEN, including the full workspace test suite, Clippy, and rustdoc; hosted negative rootless/AppArmor also passed. + +The same exact branch-coverage lane found one direct owned-production admission gap in the new helper: `src/infrastructure/podman.rs` line 889 and four short-circuit branches represented timeout/socket-configuration/write setup failures. Functions were `190/190`, while the helper left total lines `1989/1990`, canonical regions `2664/2665`, and branches `462/466`. + +Commit `236d1a67eb90f6d7c10c4714dd2d1d55faee72c6` is the minimum causal repair. It composes read-timeout, write-timeout, request-write, and response-read operations as one ordered `io::Result` chain and returns readiness only when the complete chain succeeds with an HTTP/1.1 2xx prefix. This is not a coverage-threshold exception: every setup/I/O error remains fail closed, but the implementation no longer duplicates one semantic failure outcome across four explicit short-circuit branches. Rust's standard `TcpStream` contract already rejects a zero `Duration` for both read and write timeouts, so removing the separate `timeout.is_zero()` predicate does not create an unbounded-I/O path. + +The ordinary coverage lane on `59cc738...` separately reproduced `BackendInvocationFailed { operation: "rootless_probe" }` in `root_coverage_edges`, before its intended `network_inspect` failure. That repeated process-invocation class is owned by Issue #71 / Draft #72. #104 must not hide it with retries, mutexes, environment workarounds, or weakened expectations. + +## Review-driven protocol-integrity RED and minimum repair + +Code review of hosted-GREEN predecessor `d098385045cefd8b337ba2bd0069107a01756b81` found two acceptance defects in the probe itself. The request used a fixed `Host: 127.0.0.1` even though the actual runtime authority is the dynamically selected `127.0.0.1:`. The response check read only `HTTP/1.1 2`, so malformed status codes whose first character was `2` could be promoted to readiness. + +Test-only exact `32972162bfee95112b2f79a3427ebe4815e24b39` made both findings causal. Native CI `34267331198`, branch-coverage job `102199888859`, reached `tests/application_service_http_readiness_red.rs` and failed exactly three focused controls: the captured request omitted the selected port from `Host`, and malformed `HTTP/1.1 2x0 ...` / `HTTP/1.1 20x ...` responses both produced `Ok(ApplicationServiceLease)` instead of `ReadinessTimeout`. Existing TCP-only timeout, valid HTTP 204, and HTTP 503 controls remained GREEN in the same test binary. + +Minimum descendant `b481086cbd13a1e94cf8df09d49efb9fa3200e85` changes only `src/infrastructure/podman.rs`: + +- the fixed request constant is replaced by a request generated from the already-validated runtime-selected loopback port, producing `Host: 127.0.0.1:`; +- the probe reads a bounded 13-byte HTTP/1.1 status prefix and requires `HTTP/1.1 `, a literal `2`, two ASCII decimal digits, and the required following space; +- existing bounded read/write timeout chaining, fixed `/` path, TCP connect-only readiness, cleanup, and consumer-neutral ownership remain unchanged. + +RFC 9110 section 7.2 defines `Host = uri-host [ ":" port ]` as target authority information. RFC 9112 requires an HTTP/1.1 `Host` field consistent with the target URI authority and defines the status line as HTTP-version, space, a three-digit status code, space, and optional reason phrase. The repair therefore tightens the probe to the protocol grammar without adding application authentication semantics. + +## DDD and security decision + +`application_service` owns the consumer-neutral meaning of an HTTP-ready leased service. `sandbox_execution` owns bounded readiness/lifecycle and cleanup truth. `infrastructure` performs concrete socket/HTTP I/O against only the runtime-generated loopback endpoint. Podman, Docker/Colima-compatible OCI execution, gVisor/containerd, or Kubernetes adapters must not alter the readiness domain contract merely because the backend changes. + +An arbitrary origin is not accepted. The probe target is derived from the already-validated runtime-owned loopback endpoint. Application-specific authentication semantics remain typed Supporting-context configuration or consumer ACL data rather than Core UL. + +RFC 9110 defines 2xx as the successful response class. The P0 probe uses only that status class rather than embedding application-specific response bodies or login semantics. The current wire shape therefore remains unchanged. + +## Evidence and release gates + +This lane is not release authority until its final exact head reacquires repository validation, fmt/test/Clippy/rustdoc, 100% owned-production statement/function/source-region/branch coverage, hosted negative and real positive effective-LSM evidence where applicable, review/security gates, protected integration, SBOM/provenance/reproducibility/rollback, and immutable publication before any consumer version bump. + +## References + +Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics (RFC 9110).* RFC Editor. https://www.rfc-editor.org/rfc/rfc9110.html + +Thomson, M., & Nottingham, M. (2022). *HTTP/1.1 (RFC 9112).* RFC Editor. https://www.rfc-editor.org/rfc/rfc9112.html + +Rust Project Developers. (2026). *TcpStream in std::net*. Rust standard library documentation. https://doc.rust-lang.org/std/net/struct.TcpStream.html + +Souppaya, M., Morello, J., & Scarfone, K. (2017). *Application container security guide* (NIST Special Publication 800-190). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-190 \ No newline at end of file diff --git a/src/infrastructure/podman.rs b/src/infrastructure/podman.rs index f606fe00..b162eae9 100644 --- a/src/infrastructure/podman.rs +++ b/src/infrastructure/podman.rs @@ -1,6 +1,7 @@ //! Rootless Podman infrastructure adapter for isolated application services. use std::{ + io::{Read, Write}, net::{Ipv4Addr, SocketAddrV4, TcpStream}, path::PathBuf, process::Output, @@ -14,7 +15,7 @@ use sha2::{Digest, Sha256}; use super::bounded_command::{BoundedCommandError, BoundedCommandRunner}; use crate::{ ApplicationServiceError, ApplicationServiceLease, ApplicationServiceRequest, CleanupReceipt, - IsolationPolicy, ServiceEndpoint, sandbox_execution::RuntimeLeaseMetadata, + IsolationPolicy, ServiceEndpoint, ServiceProtocol, sandbox_execution::RuntimeLeaseMetadata, }; const PODMAN_BACKEND_ID: &str = "rootless_podman"; @@ -365,7 +366,7 @@ impl RootlessPodmanAdapter { } }; - if wait_for_readiness(host_port, policy).is_err() { + if wait_for_readiness(host_port, request.protocol, policy).is_err() { self.cleanup_started_container(&plan, policy.shutdown_grace_seconds)?; return Err(ApplicationServiceError::ReadinessTimeout); } @@ -851,6 +852,7 @@ fn parse_loopback_port(stdout: &[u8]) -> Option { fn wait_for_readiness( host_port: u16, + protocol: ServiceProtocol, policy: &IsolationPolicy, ) -> Result<(), ApplicationServiceError> { let deadline = Instant::now() + Duration::from_millis(policy.readiness_timeout_millis); @@ -862,10 +864,34 @@ fn wait_for_readiness( return Err(ApplicationServiceError::ReadinessTimeout); } let remaining = deadline.saturating_duration_since(now); - if TcpStream::connect_timeout(&address.into(), poll.min(remaining)).is_ok() { - return Ok(()); + if let Ok(mut stream) = TcpStream::connect_timeout(&address.into(), poll.min(remaining)) { + if protocol == ServiceProtocol::Tcp { + return Ok(()); + } + let probe_timeout = poll.min(deadline.saturating_duration_since(Instant::now())); + if http_response_is_ready(&mut stream, host_port, probe_timeout) { + return Ok(()); + } } let after_probe = Instant::now(); thread::sleep(poll.min(deadline.saturating_duration_since(after_probe))); } } + +fn http_response_is_ready(stream: &mut TcpStream, host_port: u16, timeout: Duration) -> bool { + let request = + format!("GET / HTTP/1.1\r\nHost: 127.0.0.1:{host_port}\r\nConnection: close\r\n\r\n"); + let mut status_prefix = [0_u8; 13]; + stream + .set_read_timeout(Some(timeout)) + .and_then(|()| stream.set_write_timeout(Some(timeout))) + .and_then(|()| stream.write_all(request.as_bytes())) + .and_then(|()| stream.read_exact(&mut status_prefix)) + .is_ok_and(|()| { + status_prefix.starts_with(b"HTTP/1.1 ") + && status_prefix[9] == b'2' + && status_prefix[10].is_ascii_digit() + && status_prefix[11].is_ascii_digit() + && status_prefix[12] == b' ' + }) +} diff --git a/tests/application_service_http_readiness_red.rs b/tests/application_service_http_readiness_red.rs new file mode 100644 index 00000000..8d4de9c3 --- /dev/null +++ b/tests/application_service_http_readiness_red.rs @@ -0,0 +1,277 @@ +//! Protocol-level HTTP readiness tests for the application-service boundary. + +#![cfg(target_os = "linux")] + +use std::{ + fs, + io::{Read, Write}, + net::TcpListener, + os::unix::fs::PermissionsExt, + path::PathBuf, + sync::atomic::{AtomicU64, Ordering}, + thread, + time::{SystemTime, UNIX_EPOCH}, +}; + +use quarantine_sandbox_runtime::{ + ApplicationServiceError, ApplicationServiceRequest, IsolationPolicy, ResourceRequest, + RootlessPodmanAdapter, ServiceProtocol, +}; + +static NEXT_TEMP_PATH_ID: AtomicU64 = AtomicU64::new(0); + +fn temporary_path(name: &str) -> PathBuf { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("system clock should be after the Unix epoch") + .as_nanos(); + let unique_id = NEXT_TEMP_PATH_ID.fetch_add(1, Ordering::Relaxed); + std::env::temp_dir().join(format!( + "quarantine-sandbox-runtime-http-readiness-{name}-{}-{nanos}-{unique_id}", + std::process::id() + )) +} + +fn digest_image() -> String { + format!("localhost/cwl/caido@sha256:{}", "b".repeat(64)) +} + +fn policy() -> IsolationPolicy { + IsolationPolicy { + policy_id: "http_readiness_policy_v1".to_owned(), + maximum_memory_bytes: 512 * 1024 * 1024, + maximum_cpu_millicores: 2_000, + maximum_processes: 128, + maximum_lease_seconds: 900, + maximum_tmpfs_bytes: 128 * 1024 * 1024, + readiness_timeout_millis: 80, + readiness_poll_interval_millis: 10, + shutdown_grace_seconds: 2, + run_as_user_id: 65_532, + run_as_group_id: 65_532, + } +} + +fn request() -> ApplicationServiceRequest { + ApplicationServiceRequest { + schema_version: "1.0.0".to_owned(), + request_id: "http_readiness_request".to_owned(), + image_reference: digest_image(), + container_port: 8_080, + protocol: ServiceProtocol::Http, + command: vec!["serve".to_owned()], + resources: ResourceRequest { + memory_bytes: 256 * 1024 * 1024, + cpu_millicores: 1_000, + maximum_processes: 32, + lease_seconds: 300, + tmpfs_bytes: 32 * 1024 * 1024, + }, + } +} + +fn write_fake_podman(ready_port: u16) -> (PathBuf, PathBuf) { + let program = temporary_path("fake-podman"); + let log = temporary_path("fake-podman-log"); + let info = r#"{"host":{"security":{"rootless":true,"seccompEnabled":true,"seccompProfilePath":"/usr/share/containers/seccomp.json","apparmorEnabled":true,"selinuxEnabled":false}}}"#; + let container = r#"[{"Id":"fake-container-id","AppArmorProfile":"containers-default","ProcessLabel":"","EffectiveCaps":[],"BoundingCaps":[],"Config":{"User":"65532:65532"},"HostConfig":{"ReadonlyRootfs":true,"Privileged":false,"SecurityOpt":["no-new-privileges"],"UsernsMode":"auto","PidMode":"private","IpcMode":"none","Memory":268435456,"NanoCpus":1000000000,"PidsLimit":32}}]"#; + let network = r#"[{"internal":true,"dns_enabled":false}]"#; + let script = format!( + "#!/bin/sh\nset -eu\nprintf '%s\\n' \"$*\" >> '{}'\nif [ \"${{1:-}}\" = info ]; then\n if [ \"${{3:-}}\" = json ]; then printf '%s\\n' '{}'; else printf 'true\\n'; fi\n exit 0\nfi\ncase \"${{1:-}}:${{2:-}}\" in\n network:create) : ;;\n network:inspect) printf '%s\\n' '{}' ;;\n network:rm) : ;;\n container:inspect) printf '%s\\n' '{}' ;;\n create:--name) printf 'fake-container-id\\n' ;;\n start:*) : ;;\n top:*) printf 'PID SECCOMP CAPEFF CAPBND CAPINH CAPPRM CAPAMB LABEL\\n1 filter - - - - - containers-default (enforce)\\n' ;;\n port:*) printf '127.0.0.1:{ready_port}\\n' ;;\n stop:*) : ;;\n rm:*) : ;;\n *) exit 91 ;;\nesac\n", + log.display(), + info, + network, + container, + ); + fs::write(&program, script).expect("fake Podman should be writable"); + let mut permissions = fs::metadata(&program) + .expect("fake Podman metadata should exist") + .permissions(); + permissions.set_mode(0o700); + fs::set_permissions(&program, permissions).expect("fake Podman should be executable"); + (program, log) +} + +fn remove_fixture(program: PathBuf, log: PathBuf) { + let _ = fs::remove_file(program); + let _ = fs::remove_file(log); +} + +fn spawn_http_response(listener: TcpListener, response: &'static [u8]) -> thread::JoinHandle<()> { + thread::spawn(move || { + let (mut stream, _) = listener.accept().expect("readiness probe should connect"); + stream + .write_all(response) + .expect("HTTP response should be writable"); + }) +} + +fn spawn_http_response_capturing_request( + listener: TcpListener, + response: &'static [u8], +) -> thread::JoinHandle> { + thread::spawn(move || { + let (mut stream, _) = listener.accept().expect("readiness probe should connect"); + let mut request = Vec::new(); + let mut chunk = [0_u8; 128]; + while !request.ends_with(b"\r\n\r\n") { + let bytes_read = stream + .read(&mut chunk) + .expect("HTTP readiness request should be readable"); + if bytes_read == 0 { + break; + } + request.extend_from_slice(&chunk[..bytes_read]); + assert!( + request.len() <= 1_024, + "HTTP readiness request must stay bounded" + ); + } + stream + .write_all(response) + .expect("HTTP response should be writable"); + request + }) +} + +fn assert_malformed_http_status_is_not_ready(response: &'static [u8]) { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("listener should expose its address") + .port(); + let responder = spawn_http_response(listener, response); + let (program, log) = write_fake_podman(ready_port); + let adapter = RootlessPodmanAdapter::new(program.clone()); + + assert_eq!( + adapter.launch_at(&request(), &policy(), 1_780_000_000), + Err(ApplicationServiceError::ReadinessTimeout) + ); + responder.join().expect("HTTP responder should finish"); + remove_fixture(program, log); +} + +#[test] +fn http_service_does_not_become_ready_from_tcp_acceptance_alone() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("listener should expose its address") + .port(); + let (program, log) = write_fake_podman(ready_port); + let adapter = RootlessPodmanAdapter::new(program.clone()); + + let result = adapter.launch_at(&request(), &policy(), 1_780_000_000); + let calls = fs::read_to_string(&log).expect("fake Podman calls should be recorded"); + remove_fixture(program, log); + drop(listener); + + assert_eq!(result, Err(ApplicationServiceError::ReadinessTimeout)); + assert!(calls.contains("stop --time 2")); + assert!(calls.contains("rm --force")); + assert!(calls.contains("network rm --force")); +} + +#[test] +fn http_service_becomes_ready_after_a_bounded_success_response() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("listener should expose its address") + .port(); + let responder = spawn_http_response( + listener, + b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\nConnection: close\r\n\r\n", + ); + let (program, log) = write_fake_podman(ready_port); + let adapter = RootlessPodmanAdapter::new(program.clone()); + + let lease = adapter + .launch_at(&request(), &policy(), 1_780_000_000) + .expect("valid HTTP response should establish protocol readiness"); + responder.join().expect("HTTP responder should finish"); + assert_eq!(lease.endpoint().protocol(), ServiceProtocol::Http); + assert_eq!(lease.endpoint().host(), "127.0.0.1"); + assert_eq!(lease.endpoint().port(), ready_port); + + adapter + .terminate_at(&lease, 1_780_000_001) + .expect("successful HTTP lease should remain cleanable"); + remove_fixture(program, log); +} + +#[test] +fn http_readiness_request_uses_the_selected_loopback_authority() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("listener should expose its address") + .port(); + let responder = spawn_http_response_capturing_request( + listener, + b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\nConnection: close\r\n\r\n", + ); + let (program, log) = write_fake_podman(ready_port); + let adapter = RootlessPodmanAdapter::new(program.clone()); + + let lease = adapter + .launch_at(&request(), &policy(), 1_780_000_000) + .expect("valid HTTP response should establish protocol readiness"); + let captured_request = responder.join().expect("HTTP responder should finish"); + let captured_request = std::str::from_utf8(&captured_request) + .expect("HTTP readiness request should use ASCII header syntax"); + assert!(captured_request.contains(&format!("\r\nHost: 127.0.0.1:{ready_port}\r\n"))); + + adapter + .terminate_at(&lease, 1_780_000_001) + .expect("successful HTTP lease should remain cleanable"); + remove_fixture(program, log); +} + +#[test] +fn malformed_http_protocol_prefix_is_not_readiness() { + assert_malformed_http_status_is_not_ready( + b"NOTP/1.1 204 Invalid\r\nContent-Length: 0\r\nConnection: close\r\n\r\n", + ); +} + +#[test] +fn malformed_http_status_with_invalid_second_digit_is_not_readiness() { + assert_malformed_http_status_is_not_ready( + b"HTTP/1.1 2x0 Invalid\r\nContent-Length: 0\r\nConnection: close\r\n\r\n", + ); +} + +#[test] +fn malformed_http_status_with_invalid_third_digit_is_not_readiness() { + assert_malformed_http_status_is_not_ready( + b"HTTP/1.1 20x Invalid\r\nContent-Length: 0\r\nConnection: close\r\n\r\n", + ); +} + +#[test] +fn http_server_error_is_not_readiness() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("listener should expose its address") + .port(); + let responder = spawn_http_response( + listener, + b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n", + ); + let (program, log) = write_fake_podman(ready_port); + let adapter = RootlessPodmanAdapter::new(program.clone()); + + assert_eq!( + adapter.launch_at(&request(), &policy(), 1_780_000_000), + Err(ApplicationServiceError::ReadinessTimeout) + ); + responder.join().expect("HTTP responder should finish"); + let calls = fs::read_to_string(&log).expect("cleanup calls should be recorded"); + assert!(calls.contains("stop --time 2")); + assert!(calls.contains("rm --force")); + assert!(calls.contains("network rm --force")); + remove_fixture(program, log); +} diff --git a/tests/podman_application_service.rs b/tests/podman_application_service.rs index 04647d5a..5900553d 100644 --- a/tests/podman_application_service.rs +++ b/tests/podman_application_service.rs @@ -44,7 +44,7 @@ fn request() -> ApplicationServiceRequest { request_id: "process_boundary_request".to_owned(), image_reference: digest_image(), container_port: 8_080, - protocol: ServiceProtocol::Http, + protocol: ServiceProtocol::Tcp, command: vec!["serve".to_owned()], resources: ResourceRequest { memory_bytes: 256 * 1024 * 1024, @@ -120,7 +120,7 @@ fn launch_requires_rootless_backend_and_returns_loopback_lease_then_cleans_up() assert_eq!(lease.request_id(), "process_boundary_request"); assert_eq!(lease.endpoint().host(), "127.0.0.1"); assert_eq!(lease.endpoint().port(), ready_port); - assert_eq!(lease.endpoint().protocol(), ServiceProtocol::Http); + assert_eq!(lease.endpoint().protocol(), ServiceProtocol::Tcp); assert_eq!(lease.image_reference(), digest_image()); assert_eq!(lease.backend_id(), "rootless_podman"); assert_eq!(lease.policy_id(), "process_boundary_policy_v1");