diff --git a/CHANGELOG.md b/CHANGELOG.md index b5cb71fb..48e5d1b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,8 @@ The format follows Keep a Changelog, and this project uses Semantic Versioning. - Application-service lease schema `1.1.0`; request and cleanup contracts remain `1.0.0`. - Process-boundary fake-Podman integration tests covering launch/readiness/termination and fail-closed readiness cleanup. - Real rootless-Podman acceptance covering the pinned backend, immutable fixture pre-pull, effective isolation, bounded HTTP readiness, explicit cleanup, and final container/network leak rejection on the reviewed source head. +- Caller-scoped `LeaseOwnerId`, `ApplicationServiceBackend` port, and process-local `ApplicationServiceCoordinator` for active-lease ownership, idempotent replay, bounded expiry cleanup, and backend-neutral lifecycle coordination. +- Regression coverage for duplicate retry suppression, changed-request conflicts, effective-policy conflicts, wrong-owner termination, concurrent duplicate launch, failed-launch reservation release, expired-lease attribution, and cleanup-failure fairness across more than one bounded cleanup batch. - Consumer owner-path integration issue for `contextual-orchestrator` so Chat/Agent domain code consumes the published lease contract rather than directly invoking Podman/containerd. - Architectural fitness validation for unique ADR identifiers, bounded-context dependency direction, and infrastructure-adapter placement. @@ -34,6 +36,8 @@ The format follows Keep a Changelog, and this project uses Semantic Versioning. - Product responsibility is broadened from artifact-analysis-only to reusable hostile-workload isolation plus artifact-analysis evidence while preserving consumer business authority. - Artifact-analysis implementation moved from generic crate-root files into `src/artifact_analysis/` to match the accepted DDD bounded context while preserving the public crate facade. - Rootless Podman implementation moved from the Core `sandbox_execution` path into `src/infrastructure/`; the Core no longer depends on `application_service` error types. +- Podman now implements the application-service lifecycle port from `src/infrastructure/`; the Supporting `application_service` coordinator does not depend on the concrete Podman adapter. +- Failed expired-lease cleanup now increments a bounded retry-attempt counter; later cleanup passes prioritize expired leases with fewer attempts before repeatedly failing entries, preventing the first 64 failures from starving later expired workloads. - Pre-publication duplicate ADR identifiers were consolidated into the canonical ADR 0001–0006 sequence before protected-branch integration. - Evidence identity now includes policy, source revision, and ordered analyzer identifiers. - Static analyzer findings are restricted to file-format and static-capability evidence. @@ -54,8 +58,12 @@ The format follows Keep a Changelog, and this project uses Semantic Versioning. - Podman backend must report rootless mode; service publication is validated as IPv4 loopback before a lease is returned. - Podman host proxy environment inheritance is explicitly disabled so `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` values cannot become ambient application inputs. - Partial-launch/readiness failures attempt cleanup and cleanup uncertainty becomes `CleanupFailed` rather than being hidden. +- Lease ownership is scoped by authenticated command context rather than an untrusted request field; wrong-owner cleanup fails before the backend is invoked. +- Application-service replay is bound to both immutable request content and the full effective isolation policy, so a changed policy cannot silently reuse a lease created under older limits. +- Repeated cleanup failures cannot monopolize the bounded expiry-cleanup window and indefinitely hide other expired application-service leases. ### Not yet release evidence - The real rootless-Podman lane passed on the reviewed source head, but final release readiness still requires the same acceptance to remain green on the unchanged release head together with verify, complete coverage, security, SAST, review, SBOM, provenance, and protected-merge evidence. Fake-process tests alone remain insufficient isolation proof. -- Durable orphan/lease reclamation, gVisor/containerd/Kubernetes adapters, controlled egress, secret broker, and stronger dynamic-detonation profiles remain follow-on work. +- Caller-scoped lease ownership is currently process-local; authenticated transport binding, durable restart/orphan reclamation, distributed admission/resource reservation, stable wire errors, and signed durable receipts remain follow-on work. +- gVisor/containerd/Kubernetes adapters, controlled egress, secret broker, and stronger dynamic-detonation profiles remain follow-on work. diff --git a/Cargo.lock b/Cargo.lock index 38f44fb7..ad183731 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -113,21 +113,10 @@ checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" dependencies = [ "cfg-if", "libc", - "r-efi 5.3.0", + "r-efi", "wasip2", ] -[[package]] -name = "getrandom" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" -dependencies = [ - "cfg-if", - "libc", - "r-efi 6.0.0", -] - [[package]] name = "itoa" version = "1.0.18" @@ -212,6 +201,7 @@ dependencies = [ "serde", "serde_json", "sha2", + "tempfile", "thiserror", ] @@ -236,12 +226,6 @@ version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - [[package]] name = "rand" version = "0.9.5" @@ -268,7 +252,7 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" dependencies = [ - "getrandom 0.3.4", + "getrandom", ] [[package]] @@ -394,7 +378,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.4.3", + "getrandom", "once_cell", "rustix", "windows-sys", diff --git a/Cargo.toml b/Cargo.toml index e5dbab68..1e85079b 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -22,6 +22,7 @@ thiserror = "2.0" [dev-dependencies] proptest = "1.6" +tempfile = "3.27" [lints.rust] missing_docs = "deny" diff --git a/docs/doctoring/APPLICATION_SERVICE_LEASE_OWNERSHIP_GAP_OWNER_REPAIR.md b/docs/doctoring/APPLICATION_SERVICE_LEASE_OWNERSHIP_GAP_OWNER_REPAIR.md new file mode 100644 index 00000000..2bfb09ae --- /dev/null +++ b/docs/doctoring/APPLICATION_SERVICE_LEASE_OWNERSHIP_GAP_OWNER_REPAIR.md @@ -0,0 +1,35 @@ +# Application-service lease ownership gap-owner repair + +## Scope + +PR #6 owns caller-scoped application-service lease ownership and idempotency in the Supporting `application_service` bounded context. `LeaseOwnerId`, `ApplicationServiceBackend`, `ApplicationServiceCoordinator`, active-lease ownership, idempotent replay, bounded expiry cleanup, and cleanup fairness remain #6 domain truth. Backend invocation identity (#20/#40), consumer authentication, durable restart/orphan recovery, and Core sandbox-isolation semantics remain separate owner responsibilities. + +`docs/product-technical-gap-baseline.md` is repository-wide Gap authority and is maintained by canonical PR #121. A focused application-service branch must not retain its own live copy of that ledger. + +## Retained causal evidence + +Historical exact #6 head `8daab565bba1b61e605488c080eb28de371990a3` executed native CI `33962779845`. Verify `101297469596` passed the full test/lint/doc path and hosted negative rootless/AppArmor `101297469544` passed. Coverage remained an actual admission failure: `101297469496` failed during generation and branch coverage `101297469509` reported lines `2356/2442` (96.48%), functions `218/228` (95.61%), regions `3085/3211` (96.08%), and branches `429/492` (87.20%). The report contained both child-owned coordinator gaps and inherited runtime gaps, so it is historical exact-head evidence rather than current GREEN authority. + +The #6 lineage also removed an unsupported generic `Command::spawn` retry after no focused `io::ErrorKind` evidence justified it. Root issue #24 native-CI branch-trigger evidence is inherited foundation context; it is not application-service lease-ownership domain truth. + +## Single-writer finding and decision + +Review `5230212241` found that #6 still changed the repository-wide Gap ledger against exact PR base `0f765af1a4eea83029febee3b24c55cd7e7ce4e1`. The stale delta mixed valid #6 ownership/idempotency history with inherited root state, so deleting it without migration would lose causal context. + +The repair was migration-first: + +1. preserve the #6-specific ownership/idempotency, coverage, and inherited-prerequisite distinctions in this owner-local record; +2. restore only `docs/product-technical-gap-baseline.md` byte-for-byte to the then-exact base blob `5f17a748cf92810963ea67b30ce54675a7c6d919`; +3. keep repository-wide live Gap updates in #121 rather than copying the latest #121 file into this leaf. + +No production Rust, public contract, fixture, or test semantics were changed by that ownership repair. Historical CI does not transfer to moved heads. + +## Live-root adoption repair + +Fresh review `5230412333` found that live root `feat/runtime-foundation-tdd@5c6a44bb2b35eb17d0315d72db242f4488c3c426` had advanced 33 commits beyond #6's recorded base and that a parent-tree preference would silently discard valid deltas. The independent overlaps were `.github/workflows/ci.yml`, `src/application_service/mod.rs`, and `tests/runtime_boundary_regressions.rs`. + +The repair was staged and then completed without force/rebase. Commit `a08a786e7744f6697d93da6f41a1a04796cad10e` first adopted exact root CI blob `d172e830706afc290696c818730e1cf570df2be6`, including `persist-credentials: false` on every checkout. The final ordinary two-parent commit `64283e08d99b353430bc1ce97f20019d89f8fbd0` uses the live root tree as the merge-tree foundation so every root-only coverage/runtime delta survives, while overlaying the #6-owned coordinator, backend, package and focused-test deltas. + +The two semantic overlaps were merged explicitly rather than hidden by an evil merge. `src/application_service/mod.rs` blob `74270f29a1fe60b6e3a739514a5f17ab685bb3b4` keeps the #6 coordinator module/export and the root parser/coverage simplification (`split_once`, descendant `skip(1)`, parser-dominated no-colon success). `tests/runtime_boundary_regressions.rs` blob `78b11bf8aebcd08dda86b9137963e16e2fc2e0e8` keeps #6's isolated `tempfile` fixtures while binding fake runtime identity to the root-safe 64-character lower-hex container identifier. Root-only `docs/product-technical-gap-baseline.md`, coverage scripts/tests and Podman/root coverage deltas remain inherited from `5c6a44...`; they are not re-authored as #6-owned changes. + +This makes `5c6a44bb2b35eb17d0315d72db242f4488c3c426` an actual ancestor of #6. The resulting exact head must reacquire repository, formatting, full-test, Clippy/rustdoc, complete coverage, review/security, applicable positive-isolation, protected-integration and immutable-release evidence. Predecessor GREEN never transfers. Descendants must adopt this moved parent normally and preserve any overlapping effective-isolation/runtime deltas before they are considered current. diff --git a/src/application_service/coordinator.rs b/src/application_service/coordinator.rs new file mode 100644 index 00000000..6d35ec3b --- /dev/null +++ b/src/application_service/coordinator.rs @@ -0,0 +1,740 @@ +//! Caller-scoped application-service lifecycle coordination. + +use std::{collections::BTreeMap, sync::Mutex}; + +use sha2::{Digest, Sha256}; +use thiserror::Error; + +use super::{ + ApplicationServiceError, ApplicationServiceLease, ApplicationServiceRequest, CleanupReceipt, +}; +use crate::IsolationPolicy; + +const MAX_LEASE_OWNER_ID_BYTES: usize = 128; +const MAX_EXPIRED_CLEANUPS_PER_CALL: usize = 64; + +/// Coordinator-owned caller/idempotency errors. +#[derive(Clone, Debug, Error, PartialEq, Eq)] +pub enum ApplicationServiceCoordinatorError { + /// The authenticated caller identity could not be represented safely. + #[error("invalid application-service lease owner identity")] + InvalidLeaseOwnerId, + /// The same caller/request identity was reused for different request content. + #[error("application-service idempotency conflict")] + IdempotencyConflict, + /// An identical caller/request launch is already executing. + #[error("application-service launch already in progress")] + LaunchInProgress, + /// Cleanup for the same active lease is already executing. + #[error("application-service termination already in progress")] + TerminationInProgress, + /// The caller does not own an active lease matching the supplied receipt. + #[error("application-service lease is unknown to this caller")] + UnknownLease, + /// The caller supplied a receipt that differs from the registered active lease. + #[error("application-service lease receipt does not match active registry state")] + LeaseMismatch, + /// The process-local coordinator registry could not be accessed safely. + #[error("application-service coordinator state is unavailable")] + StateUnavailable, + /// The application-service request or sandbox backend failed. + #[error(transparent)] + Backend(#[from] ApplicationServiceError), +} + +/// Opaque authenticated-caller identity used to scope leases and idempotency. +/// +/// The runtime does not authenticate this value. A transport adapter must +/// construct it only after it has verified the caller and mapped that caller to +/// a stable opaque identity. It is intentionally separate from the untrusted +/// [`ApplicationServiceRequest`] payload. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct LeaseOwnerId(String); + +impl LeaseOwnerId { + /// Validate and construct a bounded opaque lease-owner identity. + /// + /// # Errors + /// + /// Returns [`ApplicationServiceCoordinatorError::InvalidLeaseOwnerId`] for + /// empty, oversized, whitespace-bearing, non-ASCII, or unsupported identities. + pub fn new(value: &str) -> Result { + let valid = !value.is_empty() + && value.len() <= MAX_LEASE_OWNER_ID_BYTES + && value.bytes().all(|byte| { + byte.is_ascii_alphanumeric() + || matches!(byte, b'.' | b'_' | b':' | b'/' | b'@' | b'-') + }); + if !valid { + return Err(ApplicationServiceCoordinatorError::InvalidLeaseOwnerId); + } + Ok(Self(value.to_owned())) + } + + /// Return the stable opaque owner identity. + #[must_use] + pub fn as_str(&self) -> &str { + &self.0 + } +} + +/// Infrastructure port required by the application-service coordinator. +/// +/// Implementations may use Podman, containerd, gVisor, or another reviewed +/// sandbox backend. Idempotency and caller ownership stay in the Supporting +/// `application_service` bounded context rather than in an infrastructure +/// adapter. +pub trait ApplicationServiceBackend: Send + Sync { + /// Launch one validated isolated application service. + /// + /// # Errors + /// + /// Returns [`ApplicationServiceError`] when the backend cannot establish + /// the requested isolation and readiness contract. + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result; + + /// Stop one active service and remove all runtime-owned isolation resources. + /// + /// # Errors + /// + /// Returns [`ApplicationServiceError`] when cleanup cannot be proven. + fn terminate_at( + &self, + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, + ) -> Result; +} + +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +struct LeaseKey { + lease_owner_id: LeaseOwnerId, + request_id: String, +} + +impl LeaseKey { + fn new(lease_owner_id: &LeaseOwnerId, request_id: &str) -> Self { + Self { + lease_owner_id: lease_owner_id.clone(), + request_id: request_id.to_owned(), + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +enum RegistryEntry { + Launching { + request_fingerprint: String, + }, + Active { + request_fingerprint: String, + lease: ApplicationServiceLease, + cleanup_attempts: u32, + }, + Terminating { + request_fingerprint: String, + lease: ApplicationServiceLease, + cleanup_attempts: u32, + }, +} + +/// One bounded expired-lease cleanup outcome. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ExpiredLeaseCleanupResult { + lease_owner_id: LeaseOwnerId, + request_id: String, + lease: ApplicationServiceLease, + result: Result, +} + +impl ExpiredLeaseCleanupResult { + /// Return the authenticated caller identity that owned the expired lease. + #[must_use] + pub const fn lease_owner_id(&self) -> &LeaseOwnerId { + &self.lease_owner_id + } + + /// Return the caller-scoped idempotency key of the expired lease. + #[must_use] + pub fn request_id(&self) -> &str { + &self.request_id + } + + /// Return the public lease receipt that was selected for cleanup. + #[must_use] + pub const fn lease(&self) -> &ApplicationServiceLease { + &self.lease + } + + /// Return the cleanup receipt or attributable backend failure. + pub const fn result(&self) -> &Result { + &self.result + } +} + +/// In-memory application-service coordinator enforcing caller ownership and idempotency. +/// +/// This coordinator is process-local. It prevents duplicate launches and +/// cross-caller cleanup within one running runtime, but it is not durable crash +/// recovery. A later persistence slice must reconstruct and reconcile leases +/// after restart before claiming orphan recovery. +pub struct ApplicationServiceCoordinator { + backend: B, + leases: Mutex>, +} + +impl ApplicationServiceCoordinator +where + B: ApplicationServiceBackend, +{ + /// Construct a coordinator around one reviewed sandbox backend. + #[must_use] + pub fn new(backend: B) -> Self { + Self { + backend, + leases: Mutex::new(BTreeMap::new()), + } + } + + /// Launch or replay one caller-scoped application-service request. + /// + /// An identical retry for an active lease returns that lease without + /// invoking the backend again. Reusing the same caller/request identity for + /// different immutable request or effective policy content fails closed. + /// + /// # Errors + /// + /// Returns [`ApplicationServiceCoordinatorError`] for invalid requests, + /// idempotency conflicts, concurrent duplicate launches, state failure, or + /// backend isolation failures. + pub fn launch_at( + &self, + lease_owner_id: &LeaseOwnerId, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + request.validate(policy)?; + let key = LeaseKey::new(lease_owner_id, &request.request_id); + let request_fingerprint = fingerprint_request_and_policy(request, policy); + { + let mut leases = self.lock_registry()?; + match leases.get(&key) { + Some(RegistryEntry::Launching { + request_fingerprint: existing, + }) if existing == &request_fingerprint => { + return Err(ApplicationServiceCoordinatorError::LaunchInProgress); + } + Some(RegistryEntry::Active { + request_fingerprint: existing, + lease, + .. + }) if existing == &request_fingerprint => return Ok(lease.clone()), + Some(RegistryEntry::Terminating { + request_fingerprint: existing, + .. + }) if existing == &request_fingerprint => { + return Err(ApplicationServiceCoordinatorError::TerminationInProgress); + } + Some(_) => return Err(ApplicationServiceCoordinatorError::IdempotencyConflict), + None => { + leases.insert( + key.clone(), + RegistryEntry::Launching { + request_fingerprint: request_fingerprint.clone(), + }, + ); + } + } + } + + let lease = match self + .backend + .launch_at(request, policy, started_at_epoch_seconds) + { + Ok(lease) => lease, + Err(error) => { + self.lock_registry()?.remove(&key); + return Err(error.into()); + } + }; + + let mut leases = match self.lock_registry() { + Ok(leases) => leases, + Err(error) => { + self.backend + .terminate_at(&lease, started_at_epoch_seconds)?; + return Err(error); + } + }; + leases.insert( + key, + RegistryEntry::Active { + request_fingerprint, + lease: lease.clone(), + cleanup_attempts: 0, + }, + ); + Ok(lease) + } + + /// Terminate one lease only when the caller owns the active registry entry. + /// + /// Cleanup failure remains the externally visible safety result even when + /// the registry also becomes unavailable while recording that failure. + /// + /// # Errors + /// + /// Returns [`ApplicationServiceCoordinatorError::UnknownLease`] for a wrong + /// owner or unknown lease before any backend cleanup operation is attempted. + pub fn terminate_at( + &self, + lease_owner_id: &LeaseOwnerId, + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, + ) -> Result { + let key = LeaseKey::new(lease_owner_id, lease.request_id()); + let (request_fingerprint, registered_lease, cleanup_attempts) = { + let mut leases = self.lock_registry()?; + match leases.get(&key) { + Some(RegistryEntry::Active { + request_fingerprint, + lease: registered_lease, + cleanup_attempts, + }) => { + if registered_lease != lease { + return Err(ApplicationServiceCoordinatorError::LeaseMismatch); + } + let request_fingerprint = request_fingerprint.clone(); + let registered_lease = registered_lease.clone(); + let cleanup_attempts = *cleanup_attempts; + leases.insert( + key.clone(), + RegistryEntry::Terminating { + request_fingerprint: request_fingerprint.clone(), + lease: registered_lease.clone(), + cleanup_attempts, + }, + ); + (request_fingerprint, registered_lease, cleanup_attempts) + } + Some(RegistryEntry::Launching { .. }) => { + return Err(ApplicationServiceCoordinatorError::LaunchInProgress); + } + Some(RegistryEntry::Terminating { .. }) => { + return Err(ApplicationServiceCoordinatorError::TerminationInProgress); + } + None => return Err(ApplicationServiceCoordinatorError::UnknownLease), + } + }; + + let result = self + .backend + .terminate_at(®istered_lease, terminated_at_epoch_seconds); + let registry_result = self.finish_termination( + &key, + request_fingerprint, + registered_lease, + cleanup_attempts, + &result, + ); + match result { + Ok(receipt) => { + registry_result?; + Ok(receipt) + } + Err(error) => Err(error.into()), + } + } + + /// Clean up at most 64 active leases whose expiry is not later than `now`. + /// + /// Failed cleanup remains registered as active so a later operator or + /// cleanup pass can retry it. Previously unattempted expired leases are + /// selected before repeatedly failing entries so one bad cleanup cannot + /// starve later expired workloads. This process-local function does not + /// claim crash/restart orphan recovery. If both cleanup and registry + /// recording fail, the cleanup failure remains externally visible. + /// + /// # Errors + /// + /// Returns [`ApplicationServiceCoordinatorError::StateUnavailable`] if the + /// in-memory registry cannot be accessed safely while no backend cleanup + /// failure needs to take precedence. + pub fn cleanup_expired_at( + &self, + now_epoch_seconds: u64, + ) -> Result, ApplicationServiceCoordinatorError> { + let candidates = { + let mut leases = self.lock_registry()?; + let mut candidates: Vec<(LeaseKey, String, ApplicationServiceLease, u32)> = leases + .iter() + .filter_map(|(key, entry)| match entry { + RegistryEntry::Active { + request_fingerprint, + lease, + cleanup_attempts, + } if lease.expires_at_epoch_seconds() <= now_epoch_seconds => Some(( + key.clone(), + request_fingerprint.clone(), + lease.clone(), + *cleanup_attempts, + )), + _ => None, + }) + .collect(); + candidates + .sort_by(|left, right| left.3.cmp(&right.3).then_with(|| left.0.cmp(&right.0))); + candidates.truncate(MAX_EXPIRED_CLEANUPS_PER_CALL); + for (key, request_fingerprint, lease, cleanup_attempts) in &candidates { + leases.insert( + key.clone(), + RegistryEntry::Terminating { + request_fingerprint: request_fingerprint.clone(), + lease: lease.clone(), + cleanup_attempts: *cleanup_attempts, + }, + ); + } + candidates + }; + + let mut outcomes = Vec::with_capacity(candidates.len()); + for (key, request_fingerprint, lease, cleanup_attempts) in candidates { + let result = self.backend.terminate_at(&lease, now_epoch_seconds); + let registry_result = self.finish_termination( + &key, + request_fingerprint, + lease.clone(), + cleanup_attempts, + &result, + ); + if let Err(error) = &result + && registry_result.is_err() + { + return Err(error.clone().into()); + } + registry_result?; + outcomes.push(ExpiredLeaseCleanupResult { + lease_owner_id: key.lease_owner_id, + request_id: key.request_id, + lease, + result, + }); + } + Ok(outcomes) + } + + fn lock_registry( + &self, + ) -> Result< + std::sync::MutexGuard<'_, BTreeMap>, + ApplicationServiceCoordinatorError, + > { + self.leases + .lock() + .map_err(|_| ApplicationServiceCoordinatorError::StateUnavailable) + } + + fn finish_termination( + &self, + key: &LeaseKey, + request_fingerprint: String, + lease: ApplicationServiceLease, + cleanup_attempts: u32, + result: &Result, + ) -> Result<(), ApplicationServiceCoordinatorError> { + let mut leases = self.lock_registry()?; + if result.is_ok() { + leases.remove(key); + } else { + leases.insert( + key.clone(), + RegistryEntry::Active { + request_fingerprint, + lease, + cleanup_attempts: cleanup_attempts.saturating_add(1), + }, + ); + } + Ok(()) + } +} + +fn fingerprint_request_and_policy( + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, +) -> String { + let mut hasher = Sha256::new(); + for component in [ + request.schema_version.as_str(), + request.request_id.as_str(), + request.image_reference.as_str(), + request.protocol.as_str(), + policy.policy_id.as_str(), + ] { + hasher.update(component.as_bytes()); + hasher.update([0]); + } + hasher.update(request.container_port.to_be_bytes()); + for argument in &request.command { + hasher.update(argument.as_bytes()); + hasher.update([0]); + } + for value in [ + request.resources.memory_bytes, + u64::from(request.resources.cpu_millicores), + u64::from(request.resources.maximum_processes), + u64::from(request.resources.lease_seconds), + request.resources.tmpfs_bytes, + policy.maximum_memory_bytes, + u64::from(policy.maximum_cpu_millicores), + u64::from(policy.maximum_processes), + u64::from(policy.maximum_lease_seconds), + policy.maximum_tmpfs_bytes, + policy.readiness_timeout_millis, + policy.readiness_poll_interval_millis, + u64::from(policy.shutdown_grace_seconds), + u64::from(policy.run_as_user_id), + u64::from(policy.run_as_group_id), + ] { + hasher.update(value.to_be_bytes()); + } + format!("{:x}", hasher.finalize()) +} + +#[cfg(test)] +mod tests { + use std::{ + sync::{ + Arc, Barrier, + atomic::{AtomicUsize, Ordering}, + }, + thread, + }; + + use super::{ + ApplicationServiceBackend, ApplicationServiceCoordinator, + ApplicationServiceCoordinatorError, ApplicationServiceError, ApplicationServiceLease, + ApplicationServiceRequest, CleanupReceipt, LeaseOwnerId, + }; + use crate::{IsolationPolicy, ResourceRequest, ServiceEndpoint, ServiceProtocol}; + + struct BlockingBackend { + launch_entered: Arc, + launch_resume: Arc, + terminate_calls: Arc, + } + + impl ApplicationServiceBackend for BlockingBackend { + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + self.launch_entered.wait(); + self.launch_resume.wait(); + Ok(ApplicationServiceLease::new( + request, + crate::sandbox_execution::RuntimeLeaseMetadata { + backend_id: "test_backend", + sandbox_id: "sandbox-registration-gap".to_owned(), + network_id: "network-registration-gap".to_owned(), + policy_id: policy.policy_id.clone(), + policy_sha256: policy.effective_policy_sha256(), + started_at_epoch_seconds, + expires_at_epoch_seconds: started_at_epoch_seconds + + u64::from(request.resources.lease_seconds), + shutdown_grace_seconds: policy.shutdown_grace_seconds, + }, + ServiceEndpoint::loopback(45_321, request.protocol), + )) + } + + fn terminate_at( + &self, + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, + ) -> Result { + self.terminate_calls.fetch_add(1, Ordering::SeqCst); + Ok(CleanupReceipt::complete(lease, terminated_at_epoch_seconds)) + } + } + + struct TerminationFailureBackend { + terminate_entered: Arc, + terminate_resume: Arc, + } + + impl ApplicationServiceBackend for TerminationFailureBackend { + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + Ok(ApplicationServiceLease::new( + request, + crate::sandbox_execution::RuntimeLeaseMetadata { + backend_id: "test_backend", + sandbox_id: "sandbox-cleanup-failure".to_owned(), + network_id: "network-cleanup-failure".to_owned(), + policy_id: policy.policy_id.clone(), + policy_sha256: policy.effective_policy_sha256(), + started_at_epoch_seconds, + expires_at_epoch_seconds: started_at_epoch_seconds + + u64::from(request.resources.lease_seconds), + shutdown_grace_seconds: policy.shutdown_grace_seconds, + }, + ServiceEndpoint::loopback(45_322, request.protocol), + )) + } + + fn terminate_at( + &self, + _lease: &ApplicationServiceLease, + _terminated_at_epoch_seconds: u64, + ) -> Result { + self.terminate_entered.wait(); + self.terminate_resume.wait(); + Err(ApplicationServiceError::CleanupFailed) + } + } + + fn request() -> ApplicationServiceRequest { + ApplicationServiceRequest { + schema_version: "1.0.0".to_owned(), + request_id: "registration-poison".to_owned(), + image_reference: format!("localhost/cwl/tool@sha256:{}", "d".repeat(64)), + container_port: 8_080, + protocol: ServiceProtocol::Http, + command: vec!["serve".to_owned()], + resources: ResourceRequest { + memory_bytes: 64 * 1024 * 1024, + cpu_millicores: 500, + maximum_processes: 16, + lease_seconds: 30, + tmpfs_bytes: 8 * 1024 * 1024, + }, + } + } + + fn policy() -> IsolationPolicy { + IsolationPolicy { + policy_id: "registration_cleanup_policy".to_owned(), + maximum_memory_bytes: 128 * 1024 * 1024, + maximum_cpu_millicores: 1_000, + maximum_processes: 32, + maximum_lease_seconds: 60, + maximum_tmpfs_bytes: 16 * 1024 * 1024, + readiness_timeout_millis: 1_000, + readiness_poll_interval_millis: 10, + shutdown_grace_seconds: 1, + run_as_user_id: 65_532, + run_as_group_id: 65_532, + } + } + + #[test] + fn successful_backend_launch_is_cleaned_if_registry_becomes_unavailable() { + let launch_entered = Arc::new(Barrier::new(2)); + let launch_resume = Arc::new(Barrier::new(2)); + let terminate_calls = Arc::new(AtomicUsize::new(0)); + let coordinator = Arc::new(ApplicationServiceCoordinator::new(BlockingBackend { + launch_entered: Arc::clone(&launch_entered), + launch_resume: Arc::clone(&launch_resume), + terminate_calls: Arc::clone(&terminate_calls), + })); + let owner = LeaseOwnerId::new("urn:cwl:agent:test").unwrap_or_else(|_| { + std::panic::resume_unwind(Box::new( + "test owner should satisfy the bounded identity contract", + )) + }); + let worker_coordinator = Arc::clone(&coordinator); + let worker_owner = owner.clone(); + let worker = thread::spawn(move || { + worker_coordinator.launch_at(&worker_owner, &request(), &policy(), 1_780_000_000) + }); + + launch_entered.wait(); + let poison_target = Arc::clone(&coordinator); + let poison = thread::spawn(move || { + let _guard = poison_target.leases.lock().unwrap_or_else(|_| { + std::panic::resume_unwind(Box::new( + "registry should be healthy before explicit poisoning", + )) + }); + std::panic::resume_unwind(Box::new("poison registry after backend launch begins")); + }); + assert!(poison.join().is_err()); + launch_resume.wait(); + + assert_eq!( + worker.join().unwrap_or_else(|_| { + std::panic::resume_unwind(Box::new("launch worker should not panic")) + }), + Err(ApplicationServiceCoordinatorError::StateUnavailable) + ); + assert_eq!( + terminate_calls.load(Ordering::SeqCst), + 1, + "a successful backend launch must be cleaned up when the lease cannot be registered" + ); + } + + #[test] + fn cleanup_failure_is_not_hidden_by_later_registry_failure() { + let terminate_entered = Arc::new(Barrier::new(2)); + let terminate_resume = Arc::new(Barrier::new(2)); + let coordinator = Arc::new(ApplicationServiceCoordinator::new( + TerminationFailureBackend { + terminate_entered: Arc::clone(&terminate_entered), + terminate_resume: Arc::clone(&terminate_resume), + }, + )); + let owner = LeaseOwnerId::new("urn:cwl:agent:test").unwrap_or_else(|_| { + std::panic::resume_unwind(Box::new( + "test owner should satisfy the bounded identity contract", + )) + }); + let lease = coordinator + .launch_at(&owner, &request(), &policy(), 1_780_000_000) + .unwrap_or_else(|_| { + std::panic::resume_unwind(Box::new("test lease should register before termination")) + }); + let worker_coordinator = Arc::clone(&coordinator); + let worker_owner = owner.clone(); + let worker_lease = lease.clone(); + let worker = thread::spawn(move || { + worker_coordinator.terminate_at(&worker_owner, &worker_lease, 1_780_000_010) + }); + + terminate_entered.wait(); + let poison_target = Arc::clone(&coordinator); + let poison = thread::spawn(move || { + let _guard = poison_target.leases.lock().unwrap_or_else(|_| { + std::panic::resume_unwind(Box::new( + "registry should be healthy before explicit poisoning", + )) + }); + std::panic::resume_unwind(Box::new("poison registry after backend cleanup begins")); + }); + assert!(poison.join().is_err()); + terminate_resume.wait(); + + assert_eq!( + worker.join().unwrap_or_else(|_| { + std::panic::resume_unwind(Box::new("termination worker should not panic")) + }), + Err(ApplicationServiceCoordinatorError::Backend( + ApplicationServiceError::CleanupFailed, + )), + "cleanup failure must remain the externally visible safety result even when state recovery also fails" + ); + } +} diff --git a/src/application_service/mod.rs b/src/application_service/mod.rs index 14d51e04..74270f29 100644 --- a/src/application_service/mod.rs +++ b/src/application_service/mod.rs @@ -1,5 +1,12 @@ //! Supporting bounded context for launching an approved application as an isolated service. +mod coordinator; + +pub use coordinator::{ + ApplicationServiceBackend, ApplicationServiceCoordinator, ApplicationServiceCoordinatorError, + ExpiredLeaseCleanupResult, LeaseOwnerId, +}; + use serde::{Deserialize, Serialize}; use thiserror::Error; diff --git a/src/infrastructure/application_service_backend.rs b/src/infrastructure/application_service_backend.rs new file mode 100644 index 00000000..ea16d0a6 --- /dev/null +++ b/src/infrastructure/application_service_backend.rs @@ -0,0 +1,25 @@ +//! Application-service backend port implementations owned by infrastructure. + +use crate::{ + ApplicationServiceBackend, ApplicationServiceError, ApplicationServiceLease, + ApplicationServiceRequest, CleanupReceipt, IsolationPolicy, RootlessPodmanAdapter, +}; + +impl ApplicationServiceBackend for RootlessPodmanAdapter { + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + RootlessPodmanAdapter::launch_at(self, request, policy, started_at_epoch_seconds) + } + + fn terminate_at( + &self, + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, + ) -> Result { + RootlessPodmanAdapter::terminate_at(self, lease, terminated_at_epoch_seconds) + } +} diff --git a/src/infrastructure/mod.rs b/src/infrastructure/mod.rs index 5394c1be..a3db512c 100644 --- a/src/infrastructure/mod.rs +++ b/src/infrastructure/mod.rs @@ -1,5 +1,6 @@ //! Infrastructure adapters for sandbox execution. +mod application_service_backend; mod bounded_command; #[cfg(all(test, unix))] mod bounded_command_concrete_tests; diff --git a/src/lib.rs b/src/lib.rs index 1b499b7e..c60a49fc 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -14,8 +14,10 @@ mod infrastructure; mod sandbox_execution; pub use application_service::{ + ApplicationServiceBackend, ApplicationServiceCoordinator, ApplicationServiceCoordinatorError, ApplicationServiceError, ApplicationServiceLease, ApplicationServiceRequest, CleanupReceipt, - IsolationAttestation, ServiceEndpoint, ServiceProtocol, + ExpiredLeaseCleanupResult, IsolationAttestation, LeaseOwnerId, ServiceEndpoint, + ServiceProtocol, }; pub use artifact_analysis::{ AnalysisEngine, AnalysisError, AnalysisProfile, AnalysisRequest, AnalyzerFailure, diff --git a/tests/application_service_cleanup_fairness.rs b/tests/application_service_cleanup_fairness.rs new file mode 100644 index 00000000..7f760aab --- /dev/null +++ b/tests/application_service_cleanup_fairness.rs @@ -0,0 +1,149 @@ +//! Cleanup fairness regression for repeated failed application-service teardown. + +use std::sync::{Arc, Mutex}; + +use quarantine_sandbox_runtime::{ + ApplicationServiceBackend, ApplicationServiceCoordinator, ApplicationServiceError, + ApplicationServiceLease, ApplicationServiceRequest, CleanupReceipt, IsolationPolicy, + LeaseOwnerId, ResourceRequest, ServiceProtocol, +}; +use serde_json::json; + +#[derive(Clone)] +struct FailingCleanupBackend { + attempted_request_ids: Arc>>, +} + +impl ApplicationServiceBackend for FailingCleanupBackend { + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + let lease = json!({ + "schema_version": "1.1.0", + "request_id": request.request_id.clone(), + "image_reference": request.image_reference.clone(), + "backend_id": "failing_cleanup_test_backend", + "sandbox_id": format!("sandbox-{}", request.request_id), + "network_id": format!("network-{}", request.request_id), + "policy_id": policy.policy_id.clone(), + "policy_sha256": policy.effective_policy_sha256(), + "endpoint": { + "host": "127.0.0.1", + "port": 49_152, + "protocol": "http" + }, + "started_at_epoch_seconds": started_at_epoch_seconds, + "expires_at_epoch_seconds": started_at_epoch_seconds + + u64::from(request.resources.lease_seconds), + "shutdown_grace_seconds": policy.shutdown_grace_seconds, + "isolation_attestation": { + "rootless": true, + "read_only_root_filesystem": true, + "all_capabilities_dropped": true, + "no_new_privileges": true, + "isolated_user_namespace": true, + "external_egress_denied": true, + "loopback_only_publication": true, + "credentials_available": false + } + }); + serde_json::from_value(lease).map_err(|_| ApplicationServiceError::BackendCommandFailed { + operation: "test_lease_decode", + }) + } + + fn terminate_at( + &self, + lease: &ApplicationServiceLease, + _terminated_at_epoch_seconds: u64, + ) -> Result { + self.attempted_request_ids + .lock() + .expect("test attempt registry should not be poisoned") + .push(lease.request_id().to_owned()); + Err(ApplicationServiceError::CleanupFailed) + } +} + +fn policy() -> IsolationPolicy { + IsolationPolicy { + policy_id: "cleanup_fairness_policy_v1".to_owned(), + maximum_memory_bytes: 512 * 1024 * 1024, + maximum_cpu_millicores: 2_000, + maximum_processes: 128, + maximum_lease_seconds: 900, + maximum_tmpfs_bytes: 128 * 1024 * 1024, + readiness_timeout_millis: 2_000, + readiness_poll_interval_millis: 10, + shutdown_grace_seconds: 2, + run_as_user_id: 65_532, + run_as_group_id: 65_532, + } +} + +fn request(request_id: String) -> ApplicationServiceRequest { + ApplicationServiceRequest { + schema_version: "1.0.0".to_owned(), + request_id, + image_reference: format!("localhost/cwl/tool@sha256:{}", "f".repeat(64)), + container_port: 8_080, + protocol: ServiceProtocol::Http, + command: vec!["serve".to_owned()], + resources: ResourceRequest { + memory_bytes: 256 * 1024 * 1024, + cpu_millicores: 1_000, + maximum_processes: 32, + lease_seconds: 1, + tmpfs_bytes: 32 * 1024 * 1024, + }, + } +} + +#[test] +fn repeated_cleanup_failures_do_not_starve_other_expired_leases() { + let attempted_request_ids = Arc::new(Mutex::new(Vec::new())); + let backend = FailingCleanupBackend { + attempted_request_ids: Arc::clone(&attempted_request_ids), + }; + let coordinator = ApplicationServiceCoordinator::new(backend); + let owner = LeaseOwnerId::new("urn:cwl:agent:contextual-orchestrator") + .expect("test owner should satisfy the bounded contract"); + let policy = policy(); + let started_at = 1_780_000_000; + + for index in 0..65 { + coordinator + .launch_at( + &owner, + &request(format!("cleanup_request_{index:03}")), + &policy, + started_at, + ) + .expect("test backend launch should succeed"); + } + + let first_pass = coordinator + .cleanup_expired_at(started_at + 1) + .expect("first cleanup pass should access coordinator state"); + assert_eq!(first_pass.len(), 64); + assert!(first_pass.iter().all(|outcome| outcome.result().is_err())); + + let second_pass = coordinator + .cleanup_expired_at(started_at + 1) + .expect("second cleanup pass should access coordinator state"); + assert_eq!(second_pass.len(), 64); + assert!( + second_pass + .iter() + .any(|outcome| outcome.request_id() == "cleanup_request_064"), + "a previously unattempted expired lease must be selected before retrying all failed cleanup entries" + ); + + let attempts = attempted_request_ids + .lock() + .expect("test attempt registry should remain available"); + assert!(attempts.contains(&"cleanup_request_064".to_owned())); +} diff --git a/tests/application_service_coordinator_branch_coverage.rs b/tests/application_service_coordinator_branch_coverage.rs new file mode 100644 index 00000000..f73caaa5 --- /dev/null +++ b/tests/application_service_coordinator_branch_coverage.rs @@ -0,0 +1,284 @@ +//! Branch-outcome coverage for caller-scoped coordinator state transitions. + +use std::sync::{ + Arc, Barrier, + atomic::{AtomicUsize, Ordering}, +}; +use std::thread; + +use quarantine_sandbox_runtime::{ + ApplicationServiceBackend, ApplicationServiceCoordinator, ApplicationServiceCoordinatorError, + ApplicationServiceError, ApplicationServiceLease, ApplicationServiceRequest, CleanupReceipt, + IsolationPolicy, LeaseOwnerId, ResourceRequest, ServiceProtocol, +}; +use serde_json::json; + +fn owner() -> LeaseOwnerId { + LeaseOwnerId::new("urn:cwl:agent:coordinator-branch-test") + .expect("test owner identity should satisfy the bounded contract") +} + +fn policy() -> IsolationPolicy { + IsolationPolicy { + policy_id: "coordinator_branch_policy_v1".to_owned(), + maximum_memory_bytes: 512 * 1024 * 1024, + maximum_cpu_millicores: 2_000, + maximum_processes: 128, + maximum_lease_seconds: 900, + maximum_tmpfs_bytes: 128 * 1024 * 1024, + readiness_timeout_millis: 2_000, + readiness_poll_interval_millis: 10, + shutdown_grace_seconds: 2, + run_as_user_id: 65_532, + run_as_group_id: 65_532, + } +} + +fn request() -> ApplicationServiceRequest { + ApplicationServiceRequest { + schema_version: "1.0.0".to_owned(), + request_id: "coordinator_branch_request".to_owned(), + image_reference: format!("localhost/cwl/tool@sha256:{}", "d".repeat(64)), + container_port: 8_080, + protocol: ServiceProtocol::Http, + command: vec!["serve".to_owned()], + resources: ResourceRequest { + memory_bytes: 256 * 1024 * 1024, + cpu_millicores: 1_000, + maximum_processes: 32, + lease_seconds: 300, + tmpfs_bytes: 32 * 1024 * 1024, + }, + } +} + +fn lease_for( + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, +) -> ApplicationServiceLease { + serde_json::from_value(json!({ + "schema_version": "1.1.0", + "request_id": request.request_id, + "image_reference": request.image_reference, + "backend_id": "branch_test_backend", + "sandbox_id": "sandbox_branch_test", + "network_id": "network_branch_test", + "policy_id": policy.policy_id, + "policy_sha256": policy.effective_policy_sha256(), + "endpoint": { + "host": "127.0.0.1", + "port": 43123, + "protocol": "http" + }, + "started_at_epoch_seconds": started_at_epoch_seconds, + "expires_at_epoch_seconds": started_at_epoch_seconds + + u64::from(request.resources.lease_seconds), + "shutdown_grace_seconds": policy.shutdown_grace_seconds, + "isolation_attestation": { + "rootless": true, + "read_only_root_filesystem": true, + "all_capabilities_dropped": true, + "no_new_privileges": true, + "isolated_user_namespace": true, + "external_egress_denied": true, + "loopback_only_publication": true, + "credentials_available": false + } + })) + .expect("test lease should match the public serialized contract") +} + +fn cleanup_receipt( + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, +) -> CleanupReceipt { + serde_json::from_value(json!({ + "schema_version": "1.0.0", + "sandbox_id": lease.sandbox_id(), + "network_id": lease.network_id(), + "container_removed": true, + "network_removed": true, + "terminated_at_epoch_seconds": terminated_at_epoch_seconds + })) + .expect("test cleanup receipt should match the public serialized contract") +} + +#[derive(Clone)] +struct LaunchBlockingBackend { + launch_entered: Arc, + launch_release: Arc, +} + +impl ApplicationServiceBackend for LaunchBlockingBackend { + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + self.launch_entered.wait(); + self.launch_release.wait(); + Ok(lease_for(request, policy, started_at_epoch_seconds)) + } + + fn terminate_at( + &self, + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, + ) -> Result { + Ok(cleanup_receipt(lease, terminated_at_epoch_seconds)) + } +} + +#[derive(Clone)] +struct TerminationBlockingBackend { + termination_entered: Arc, + termination_release: Arc, +} + +impl ApplicationServiceBackend for TerminationBlockingBackend { + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + Ok(lease_for(request, policy, started_at_epoch_seconds)) + } + + fn terminate_at( + &self, + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, + ) -> Result { + self.termination_entered.wait(); + self.termination_release.wait(); + Ok(cleanup_receipt(lease, terminated_at_epoch_seconds)) + } +} + +#[derive(Clone)] +struct CountingBackend { + termination_calls: Arc, +} + +impl ApplicationServiceBackend for CountingBackend { + fn launch_at( + &self, + request: &ApplicationServiceRequest, + policy: &IsolationPolicy, + started_at_epoch_seconds: u64, + ) -> Result { + Ok(lease_for(request, policy, started_at_epoch_seconds)) + } + + fn terminate_at( + &self, + lease: &ApplicationServiceLease, + terminated_at_epoch_seconds: u64, + ) -> Result { + self.termination_calls.fetch_add(1, Ordering::SeqCst); + Ok(cleanup_receipt(lease, terminated_at_epoch_seconds)) + } +} + +#[test] +fn conflicting_retry_while_launching_reaches_the_idempotency_conflict_guard() { + let launch_entered = Arc::new(Barrier::new(2)); + let launch_release = Arc::new(Barrier::new(2)); + let coordinator = Arc::new(ApplicationServiceCoordinator::new(LaunchBlockingBackend { + launch_entered: Arc::clone(&launch_entered), + launch_release: Arc::clone(&launch_release), + })); + let lease_owner = owner(); + let worker_coordinator = Arc::clone(&coordinator); + let worker_owner = lease_owner.clone(); + let worker = thread::spawn(move || { + worker_coordinator.launch_at(&worker_owner, &request(), &policy(), 1_780_000_000) + }); + + launch_entered.wait(); + let mut conflicting_request = request(); + conflicting_request.command.push("--different".to_owned()); + assert_eq!( + coordinator.launch_at(&lease_owner, &conflicting_request, &policy(), 1_780_000_001,), + Err(ApplicationServiceCoordinatorError::IdempotencyConflict) + ); + + launch_release.wait(); + let lease = worker + .join() + .expect("launch thread should not panic") + .expect("original launch should complete"); + coordinator + .terminate_at(&lease_owner, &lease, 1_780_000_010) + .expect("original lease should remain terminable"); +} + +#[test] +fn terminating_entry_distinguishes_identical_retry_from_conflicting_content() { + let termination_entered = Arc::new(Barrier::new(2)); + let termination_release = Arc::new(Barrier::new(2)); + let coordinator = Arc::new(ApplicationServiceCoordinator::new( + TerminationBlockingBackend { + termination_entered: Arc::clone(&termination_entered), + termination_release: Arc::clone(&termination_release), + }, + )); + let lease_owner = owner(); + let lease = coordinator + .launch_at(&lease_owner, &request(), &policy(), 1_780_000_000) + .expect("launch should complete before termination starts"); + let worker_coordinator = Arc::clone(&coordinator); + let worker_owner = lease_owner.clone(); + let worker_lease = lease.clone(); + let worker = thread::spawn(move || { + worker_coordinator.terminate_at(&worker_owner, &worker_lease, 1_780_000_010) + }); + + termination_entered.wait(); + assert_eq!( + coordinator.launch_at(&lease_owner, &request(), &policy(), 1_780_000_011), + Err(ApplicationServiceCoordinatorError::TerminationInProgress) + ); + let mut conflicting_request = request(); + conflicting_request.command.push("--different".to_owned()); + assert_eq!( + coordinator.launch_at(&lease_owner, &conflicting_request, &policy(), 1_780_000_012,), + Err(ApplicationServiceCoordinatorError::IdempotencyConflict) + ); + + termination_release.wait(); + worker + .join() + .expect("termination thread should not panic") + .expect("original termination should complete"); +} + +#[test] +fn forged_receipt_for_known_owner_and_request_fails_before_backend_cleanup() { + let termination_calls = Arc::new(AtomicUsize::new(0)); + let coordinator = ApplicationServiceCoordinator::new(CountingBackend { + termination_calls: Arc::clone(&termination_calls), + }); + let lease_owner = owner(); + let lease = coordinator + .launch_at(&lease_owner, &request(), &policy(), 1_780_000_000) + .expect("launch should succeed"); + let mut forged_value = serde_json::to_value(&lease).expect("lease should serialize"); + forged_value["backend_id"] = json!("forged_backend"); + let forged_lease: ApplicationServiceLease = serde_json::from_value(forged_value) + .expect("forged test value should remain structurally deserializable"); + + assert_eq!( + coordinator.terminate_at(&lease_owner, &forged_lease, 1_780_000_010), + Err(ApplicationServiceCoordinatorError::LeaseMismatch) + ); + assert_eq!(termination_calls.load(Ordering::SeqCst), 0); + + coordinator + .terminate_at(&lease_owner, &lease, 1_780_000_011) + .expect("registered lease should remain terminable"); + assert_eq!(termination_calls.load(Ordering::SeqCst), 1); +} diff --git a/tests/application_service_ownership.rs b/tests/application_service_ownership.rs new file mode 100644 index 00000000..7a73c481 --- /dev/null +++ b/tests/application_service_ownership.rs @@ -0,0 +1,333 @@ +//! Caller-scoped idempotency and lease-ownership tests for isolated services. + +#![cfg(target_os = "linux")] + +use std::{ + fs, net::TcpListener, os::unix::fs::PermissionsExt, path::Path, sync::Arc, thread, + time::Duration, +}; + +use quarantine_sandbox_runtime::{ + ApplicationServiceCoordinator, ApplicationServiceCoordinatorError, ApplicationServiceError, + ApplicationServiceRequest, IsolationPolicy, LeaseOwnerId, ResourceRequest, + RootlessPodmanAdapter, ServiceProtocol, +}; + +fn digest_image() -> String { + format!("localhost/cwl/tool@sha256:{}", "d".repeat(64)) +} + +fn owner(value: &str) -> LeaseOwnerId { + LeaseOwnerId::new(value).expect("test owner identity should satisfy the bounded contract") +} + +fn policy() -> IsolationPolicy { + IsolationPolicy { + policy_id: "lease_ownership_policy_v1".to_owned(), + maximum_memory_bytes: 512 * 1024 * 1024, + maximum_cpu_millicores: 2_000, + maximum_processes: 128, + maximum_lease_seconds: 900, + maximum_tmpfs_bytes: 128 * 1024 * 1024, + readiness_timeout_millis: 2_000, + readiness_poll_interval_millis: 10, + shutdown_grace_seconds: 2, + run_as_user_id: 65_532, + run_as_group_id: 65_532, + } +} + +fn request() -> ApplicationServiceRequest { + ApplicationServiceRequest { + schema_version: "1.0.0".to_owned(), + request_id: "agent_task_lease_42".to_owned(), + image_reference: digest_image(), + container_port: 8_080, + protocol: ServiceProtocol::Http, + command: vec!["serve".to_owned()], + resources: ResourceRequest { + memory_bytes: 256 * 1024 * 1024, + cpu_millicores: 1_000, + maximum_processes: 32, + lease_seconds: 300, + tmpfs_bytes: 32 * 1024 * 1024, + }, + } +} + +fn write_fake_podman(program: &Path, log: &Path, mode: &str, ready_port: u16) { + let info = r#"{"host":{"security":{"rootless":true,"seccompEnabled":true,"seccompProfilePath":"/usr/share/containers/seccomp.json","apparmorEnabled":true,"selinuxEnabled":false}}}"#; + let container = r#"[{"Id":"fake-container-id","AppArmorProfile":"containers-default","ProcessLabel":"","EffectiveCaps":[],"BoundingCaps":[],"Config":{"User":"65532:65532"},"HostConfig":{"ReadonlyRootfs":true,"Privileged":false,"SecurityOpt":["no-new-privileges"],"UsernsMode":"auto","PidMode":"private","IpcMode":"none","Memory":268435456,"NanoCpus":1000000000,"PidsLimit":32}}]"#; + let network = r#"[{"internal":true,"dns_enabled":false}]"#; + let script = format!( + "#!/bin/sh\nset -eu\nMODE='{mode}'\nprintf '%s\\n' \"$*\" >> '{}'\nif [ \"$MODE\" = slow_rootless ] && [ \"${{1:-}}\" = info ] && [ \"${{3:-}}\" != json ]; then sleep 1; fi\nif [ \"$MODE\" = fail_rootless ] && [ \"${{1:-}}\" = info ] && [ \"${{3:-}}\" != json ]; then exit 20; fi\ncase \"${{1:-}}:${{2:-}}\" in\n info:--format) if [ \"${{3:-}}\" = json ]; then printf '%s\\n' '{}'; else printf 'true\\n'; fi ;;\n network:create) : ;;\n network:inspect) printf '%s\\n' '{}' ;;\n create:--name) printf 'fake-container-id\\n' ;;\n start:*) : ;;\n container:inspect) printf '%s\\n' '{}' ;;\n top:*) printf 'PID SECCOMP CAPEFF CAPBND CAPINH CAPPRM CAPAMB LABEL\\n1 filter - - - - - containers-default (enforce)\\n' ;;\n port:*) printf '127.0.0.1:{ready_port}\\n' ;;\n stop:*) : ;;\n rm:*) : ;;\n network:rm) : ;;\n *) exit 91 ;;\nesac\n", + log.display(), + info, + network, + container, + ); + // Never truncate an executable path that a just-finished backend process may + // still reference. Stage a complete inode and atomically replace the pathname. + let staged_program = program.with_extension("next"); + fs::write(&staged_program, script).expect("staged fake Podman should be writable"); + let mut permissions = fs::metadata(&staged_program) + .expect("staged fake Podman metadata should exist") + .permissions(); + permissions.set_mode(0o700); + fs::set_permissions(&staged_program, permissions) + .expect("staged fake Podman should be executable"); + fs::rename(&staged_program, program).expect("fake Podman replacement should be atomic"); +} + +fn count_calls(log: &Path, needle: &str) -> usize { + fs::read_to_string(log) + .unwrap_or_default() + .lines() + .filter(|line| line.contains(needle)) + .count() +} + +fn wait_until_log_contains(log: &Path, needle: &str) { + for _ in 0..100 { + if count_calls(log, needle) > 0 { + return; + } + thread::sleep(Duration::from_millis(10)); + } + panic!("timed out waiting for fake Podman call containing {needle}"); +} + +#[test] +fn lease_owner_ids_are_bounded_opaque_runtime_context() { + assert_eq!( + LeaseOwnerId::new(""), + Err(ApplicationServiceCoordinatorError::InvalidLeaseOwnerId) + ); + assert_eq!( + LeaseOwnerId::new("contains whitespace"), + Err(ApplicationServiceCoordinatorError::InvalidLeaseOwnerId) + ); + assert_eq!( + LeaseOwnerId::new(&"a".repeat(129)), + Err(ApplicationServiceCoordinatorError::InvalidLeaseOwnerId) + ); + assert_eq!( + owner("urn:cwl:agent:contextual-orchestrator").as_str(), + "urn:cwl:agent:contextual-orchestrator" + ); +} + +#[test] +fn identical_retry_returns_existing_lease_without_second_launch() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("address should resolve") + .port(); + let fixture = tempfile::tempdir().expect("isolated ownership fixture directory"); + let program = fixture.path().join("fake-podman"); + let log = fixture.path().join("fake-podman-log"); + write_fake_podman(&program, &log, "success", ready_port); + let coordinator = + ApplicationServiceCoordinator::new(RootlessPodmanAdapter::new(program.clone())); + let owner = owner("urn:cwl:agent:contextual-orchestrator"); + + let first = coordinator + .launch_at(&owner, &request(), &policy(), 1_780_000_000) + .expect("first launch should succeed"); + let retry = coordinator + .launch_at(&owner, &request(), &policy(), 1_780_000_100) + .expect("identical retry should return the active lease"); + + assert_eq!(retry, first); + assert_eq!(count_calls(&log, "network create"), 1); + coordinator + .terminate_at(&owner, &first, 1_780_000_110) + .expect("owner should terminate its lease"); + let _ = fs::remove_file(program); + let _ = fs::remove_file(log); + drop(listener); +} + +#[test] +fn same_owner_and_request_id_with_different_content_fails_closed() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("address should resolve") + .port(); + let fixture = tempfile::tempdir().expect("isolated ownership fixture directory"); + let program = fixture.path().join("fake-podman"); + let log = fixture.path().join("fake-podman-log"); + write_fake_podman(&program, &log, "success", ready_port); + let coordinator = + ApplicationServiceCoordinator::new(RootlessPodmanAdapter::new(program.clone())); + let owner = owner("urn:cwl:agent:contextual-orchestrator"); + let first = coordinator + .launch_at(&owner, &request(), &policy(), 1_780_000_000) + .expect("first launch should succeed"); + let mut changed = request(); + changed.command.push("--different".to_owned()); + + assert_eq!( + coordinator.launch_at(&owner, &changed, &policy(), 1_780_000_001), + Err(ApplicationServiceCoordinatorError::IdempotencyConflict) + ); + assert_eq!(count_calls(&log, "network create"), 1); + coordinator + .terminate_at(&owner, &first, 1_780_000_010) + .expect("original owner should still terminate the first lease"); + let _ = fs::remove_file(program); + let _ = fs::remove_file(log); + drop(listener); +} + +#[test] +fn wrong_owner_cannot_terminate_another_callers_lease() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("address should resolve") + .port(); + let fixture = tempfile::tempdir().expect("isolated ownership fixture directory"); + let program = fixture.path().join("fake-podman"); + let log = fixture.path().join("fake-podman-log"); + write_fake_podman(&program, &log, "success", ready_port); + let coordinator = + ApplicationServiceCoordinator::new(RootlessPodmanAdapter::new(program.clone())); + let correct_owner = owner("urn:cwl:agent:contextual-orchestrator"); + let wrong_owner = owner("urn:cwl:consumer:wardnet"); + let lease = coordinator + .launch_at(&correct_owner, &request(), &policy(), 1_780_000_000) + .expect("launch should succeed"); + + assert_eq!( + coordinator.terminate_at(&wrong_owner, &lease, 1_780_000_010), + Err(ApplicationServiceCoordinatorError::UnknownLease) + ); + assert_eq!(count_calls(&log, "stop --time"), 0); + coordinator + .terminate_at(&correct_owner, &lease, 1_780_000_011) + .expect("correct owner should terminate the lease"); + assert_eq!(count_calls(&log, "stop --time"), 1); + let _ = fs::remove_file(program); + let _ = fs::remove_file(log); + drop(listener); +} + +#[test] +fn failed_launch_releases_idempotency_reservation_for_retry() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("address should resolve") + .port(); + let fixture = tempfile::tempdir().expect("isolated ownership fixture directory"); + let program = fixture.path().join("fake-podman"); + let log = fixture.path().join("fake-podman-log"); + write_fake_podman(&program, &log, "fail_rootless", ready_port); + let coordinator = + ApplicationServiceCoordinator::new(RootlessPodmanAdapter::new(program.clone())); + let owner = owner("urn:cwl:agent:contextual-orchestrator"); + + assert_eq!( + coordinator.launch_at(&owner, &request(), &policy(), 1_780_000_000), + Err(ApplicationServiceCoordinatorError::Backend( + ApplicationServiceError::BackendCommandFailed { + operation: "rootless_probe", + } + )) + ); + write_fake_podman(&program, &log, "success", ready_port); + let lease = coordinator + .launch_at(&owner, &request(), &policy(), 1_780_000_001) + .expect("retry after a failed launch should be allowed"); + coordinator + .terminate_at(&owner, &lease, 1_780_000_010) + .expect("retried lease should terminate cleanly"); + let _ = fs::remove_file(program); + let _ = fs::remove_file(log); + drop(listener); +} + +#[test] +fn concurrent_duplicate_launch_is_rejected_while_first_launch_is_in_flight() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("address should resolve") + .port(); + let fixture = tempfile::tempdir().expect("isolated ownership fixture directory"); + let program = fixture.path().join("fake-podman"); + let log = fixture.path().join("fake-podman-log"); + write_fake_podman(&program, &log, "slow_rootless", ready_port); + let coordinator = Arc::new(ApplicationServiceCoordinator::new( + RootlessPodmanAdapter::new(program.clone()), + )); + let owner = owner("urn:cwl:agent:contextual-orchestrator"); + let worker_coordinator = Arc::clone(&coordinator); + let worker_owner = owner.clone(); + let worker = thread::spawn(move || { + worker_coordinator.launch_at(&worker_owner, &request(), &policy(), 1_780_000_000) + }); + wait_until_log_contains(&log, "info --format"); + + assert_eq!( + coordinator.launch_at(&owner, &request(), &policy(), 1_780_000_001), + Err(ApplicationServiceCoordinatorError::LaunchInProgress) + ); + let lease = worker + .join() + .expect("launch thread should not panic") + .expect("first launch should finish successfully"); + assert_eq!(count_calls(&log, "network create"), 1); + coordinator + .terminate_at(&owner, &lease, 1_780_000_010) + .expect("owner should terminate the finished lease"); + let _ = fs::remove_file(program); + let _ = fs::remove_file(log); + drop(listener); +} + +#[test] +fn expired_lease_cleanup_is_bounded_and_attributed_to_owner_and_request() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("address should resolve") + .port(); + let fixture = tempfile::tempdir().expect("isolated ownership fixture directory"); + let program = fixture.path().join("fake-podman"); + let log = fixture.path().join("fake-podman-log"); + write_fake_podman(&program, &log, "success", ready_port); + let coordinator = + ApplicationServiceCoordinator::new(RootlessPodmanAdapter::new(program.clone())); + let owner = owner("urn:cwl:agent:contextual-orchestrator"); + let mut short_request = request(); + short_request.resources.lease_seconds = 1; + coordinator + .launch_at(&owner, &short_request, &policy(), 1_780_000_000) + .expect("short lease should launch"); + + assert!( + coordinator + .cleanup_expired_at(1_780_000_000) + .expect("registry should be available") + .is_empty() + ); + let outcomes = coordinator + .cleanup_expired_at(1_780_000_001) + .expect("expired cleanup should access registry"); + assert_eq!(outcomes.len(), 1); + assert_eq!(outcomes[0].lease_owner_id().as_str(), owner.as_str()); + assert_eq!(outcomes[0].request_id(), "agent_task_lease_42"); + assert!(outcomes[0].result().is_ok()); + assert_eq!(count_calls(&log, "stop --time"), 1); + assert_eq!( + coordinator.terminate_at(&owner, outcomes[0].lease(), 1_780_000_002), + Err(ApplicationServiceCoordinatorError::UnknownLease) + ); + let _ = fs::remove_file(program); + let _ = fs::remove_file(log); + drop(listener); +} diff --git a/tests/application_service_policy_idempotency.rs b/tests/application_service_policy_idempotency.rs new file mode 100644 index 00000000..79b6683a --- /dev/null +++ b/tests/application_service_policy_idempotency.rs @@ -0,0 +1,109 @@ +//! Regression test that idempotency cannot replay a lease under a changed isolation policy. + +#![cfg(target_os = "linux")] + +use std::{ + fs, + net::TcpListener, + os::unix::fs::PermissionsExt, + path::PathBuf, + time::{SystemTime, UNIX_EPOCH}, +}; + +use quarantine_sandbox_runtime::{ + ApplicationServiceCoordinator, ApplicationServiceCoordinatorError, ApplicationServiceRequest, + IsolationPolicy, LeaseOwnerId, ResourceRequest, RootlessPodmanAdapter, ServiceProtocol, +}; + +fn temporary_path(name: &str) -> PathBuf { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("system clock should be after Unix epoch") + .as_nanos(); + std::env::temp_dir().join(format!( + "qsr-policy-idempotency-{name}-{}-{nanos}", + std::process::id() + )) +} + +fn policy() -> IsolationPolicy { + IsolationPolicy { + policy_id: "agent_application_policy_v1".to_owned(), + maximum_memory_bytes: 512 * 1024 * 1024, + maximum_cpu_millicores: 2_000, + maximum_processes: 128, + maximum_lease_seconds: 900, + maximum_tmpfs_bytes: 128 * 1024 * 1024, + readiness_timeout_millis: 500, + readiness_poll_interval_millis: 10, + shutdown_grace_seconds: 2, + run_as_user_id: 65_532, + run_as_group_id: 65_532, + } +} + +fn request() -> ApplicationServiceRequest { + ApplicationServiceRequest { + schema_version: "1.0.0".to_owned(), + request_id: "same_task_same_request".to_owned(), + image_reference: format!("localhost/cwl/tool@sha256:{}", "e".repeat(64)), + container_port: 8_080, + protocol: ServiceProtocol::Http, + command: vec!["serve".to_owned()], + resources: ResourceRequest { + memory_bytes: 256 * 1024 * 1024, + cpu_millicores: 1_000, + maximum_processes: 32, + lease_seconds: 300, + tmpfs_bytes: 32 * 1024 * 1024, + }, + } +} + +fn write_fake_podman(program: &PathBuf, ready_port: u16) { + let info = r#"{"host":{"security":{"rootless":true,"seccompEnabled":true,"seccompProfilePath":"/usr/share/containers/seccomp.json","apparmorEnabled":true,"selinuxEnabled":false}}}"#; + let container = r#"[{"Id":"fake-container-id","AppArmorProfile":"containers-default","ProcessLabel":"","EffectiveCaps":[],"BoundingCaps":[],"Config":{"User":"65532:65532"},"HostConfig":{"ReadonlyRootfs":true,"Privileged":false,"SecurityOpt":["no-new-privileges"],"UsernsMode":"auto","PidMode":"private","IpcMode":"none","Memory":268435456,"NanoCpus":1000000000,"PidsLimit":32}}]"#; + let network = r#"[{"internal":true,"dns_enabled":false}]"#; + let script = format!( + "#!/bin/sh\nset -eu\ncase \"${{1:-}}:${{2:-}}\" in\n info:--format) if [ \"${{3:-}}\" = json ]; then printf '%s\\n' '{}'; else printf 'true\\n'; fi ;;\n network:create) : ;;\n network:inspect) printf '%s\\n' '{}' ;;\n create:--name) printf 'fake-container-id\\n' ;;\n start:*) : ;;\n container:inspect) printf '%s\\n' '{}' ;;\n top:*) printf 'PID SECCOMP CAPEFF CAPBND CAPINH CAPPRM CAPAMB LABEL\\n1 filter - - - - - containers-default (enforce)\\n' ;;\n port:*) printf '127.0.0.1:{ready_port}\\n' ;;\n stop:*) : ;;\n rm:*) : ;;\n network:rm) : ;;\n *) exit 91 ;;\nesac\n", + info, network, container, + ); + fs::write(program, script).expect("fake Podman should be writable"); + let mut permissions = fs::metadata(program) + .expect("fake Podman metadata should exist") + .permissions(); + permissions.set_mode(0o700); + fs::set_permissions(program, permissions).expect("fake Podman should be executable"); +} + +#[test] +fn identical_request_does_not_reuse_lease_when_effective_policy_changes() { + let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); + let ready_port = listener + .local_addr() + .expect("address should resolve") + .port(); + let program = temporary_path("fake-podman"); + write_fake_podman(&program, ready_port); + let coordinator = + ApplicationServiceCoordinator::new(RootlessPodmanAdapter::new(program.clone())); + let owner = LeaseOwnerId::new("urn:cwl:agent:contextual-orchestrator") + .expect("opaque owner should validate"); + let initial_policy = policy(); + let lease = coordinator + .launch_at(&owner, &request(), &initial_policy, 1_780_000_000) + .expect("first launch should succeed"); + + let mut changed_policy = initial_policy; + changed_policy.maximum_memory_bytes += 1; + assert_eq!( + coordinator.launch_at(&owner, &request(), &changed_policy, 1_780_000_001), + Err(ApplicationServiceCoordinatorError::IdempotencyConflict) + ); + + coordinator + .terminate_at(&owner, &lease, 1_780_000_010) + .expect("original lease should still clean up"); + let _ = fs::remove_file(program); + drop(listener); +} diff --git a/tests/runtime_boundary_regressions.rs b/tests/runtime_boundary_regressions.rs index 799c1ded..78b11bf8 100644 --- a/tests/runtime_boundary_regressions.rs +++ b/tests/runtime_boundary_regressions.rs @@ -6,8 +6,8 @@ use std::{ fs, net::TcpListener, os::unix::fs::PermissionsExt, - path::PathBuf, - time::{Duration, SystemTime, UNIX_EPOCH}, + path::{Path, PathBuf}, + time::Duration, }; use quarantine_sandbox_runtime::{ @@ -16,6 +16,9 @@ use quarantine_sandbox_runtime::{ }; const FAKE_CONTAINER_ID: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; +const SECURITY_INFO: &str = r#"{"host":{"security":{"rootless":true,"seccompEnabled":true,"seccompProfilePath":"/usr/share/containers/seccomp.json","apparmorEnabled":true,"selinuxEnabled":false}}}"#; +const CONTAINER_INSPECTION: &str = r#"[{"Id":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","AppArmorProfile":"containers-default","ProcessLabel":"","EffectiveCaps":[],"BoundingCaps":[],"Config":{"User":"65532:65532"},"HostConfig":{"ReadonlyRootfs":true,"Privileged":false,"SecurityOpt":["no-new-privileges"],"UsernsMode":"auto","PidMode":"private","IpcMode":"none","Memory":268435456,"NanoCpus":1000000000,"PidsLimit":32}}]"#; +const NETWORK_INSPECTION: &str = r#"[{"internal":true,"dns_enabled":false}]"#; fn policy() -> IsolationPolicy { IsolationPolicy { @@ -51,35 +54,8 @@ fn request(digest: &str) -> ApplicationServiceRequest { } } -fn backend_info_json() -> &'static str { - r#"{"host":{"security":{"rootless":true,"seccompEnabled":true,"seccompProfilePath":"/usr/share/containers/seccomp.json","apparmorEnabled":true,"selinuxEnabled":false}}}"# -} - -fn container_inspection_json() -> &'static str { - r#"[{"Id":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","AppArmorProfile":"containers-default","ProcessLabel":"","EffectiveCaps":[],"BoundingCaps":[],"Config":{"User":"65532:65532"},"HostConfig":{"ReadonlyRootfs":true,"Privileged":false,"SecurityOpt":["no-new-privileges"],"UsernsMode":"auto","PidMode":"private","IpcMode":"none","Memory":268435456,"NanoCpus":1000000000,"PidsLimit":32}}]"# -} - -fn network_inspection_json() -> &'static str { - r#"[{"internal":true,"dns_enabled":false}]"# -} - -fn process_security_output() -> &'static str { - "PID SECCOMP CAPEFF CAPBND CAPINH CAPPRM CAPAMB LABEL\n1 filter - - - - - containers-default (enforce)\n" -} - -fn temporary_path(name: &str) -> PathBuf { - let nanos = SystemTime::now() - .duration_since(UNIX_EPOCH) - .expect("system clock should be after the Unix epoch") - .as_nanos(); - std::env::temp_dir().join(format!( - "quarantine-sandbox-runtime-{name}-{}-{nanos}", - std::process::id() - )) -} - -fn write_executable(name: &str, script: &str) -> PathBuf { - let program = temporary_path(name); +fn write_executable(fixture_directory: &Path, name: &str, script: &str) -> PathBuf { + let program = fixture_directory.join(name); fs::write(&program, script).expect("fake runtime executable should be writable"); let mut permissions = fs::metadata(&program) .expect("fake runtime executable metadata should exist") @@ -97,11 +73,14 @@ fn digest_pinned_image_accepts_numeric_sha256() { #[test] fn slow_successful_backend_command_is_polled_until_exit() { - let script = format!( - "#!/bin/sh\nset -eu\nif [ \"${{1:-}}\" = info ]; then\n sleep 0.03\n if [ \"${{3:-}}\" = json ]; then printf '%s\\n' '{}'; else printf 'true\\n'; fi\n exit 0\nfi\ncase \"${{1:-}}:${{2:-}}\" in\n network:create) exit 21 ;;\n *) exit 91 ;;\nesac\n", - backend_info_json() + let fixture = tempfile::tempdir().expect("isolated runtime boundary fixture directory"); + let program = write_executable( + fixture.path(), + "slow-podman", + &format!( + "#!/bin/sh\nset -eu\ncase \"${{1:-}}:${{2:-}}\" in\n info:--format) if [ \"${{3:-}}\" = json ]; then printf '%s\\n' '{SECURITY_INFO}'; else printf 'true\\n'; fi ;;\n network:create) sleep 0.03; exit 21 ;;\n *) exit 91 ;;\nesac\n" + ), ); - let program = write_executable("slow-podman", &script); let adapter = RootlessPodmanAdapter::new(program.clone()) .with_command_timeout(Duration::from_millis(200)); @@ -117,17 +96,13 @@ fn slow_successful_backend_command_is_polled_until_exit() { #[test] fn non_utf8_port_output_fails_closed_and_cleans_every_created_resource() { - let log = temporary_path("non-utf8-podman-log"); + let fixture = tempfile::tempdir().expect("isolated runtime boundary fixture directory"); + let log = fixture.path().join("non-utf8-podman-log"); let script = format!( - "#!/bin/sh\nset -eu\nprintf '%s\\n' \"$*\" >> '{}'\nif [ \"${{1:-}}\" = info ]; then\n if [ \"${{3:-}}\" = json ]; then printf '%s\\n' '{}'; else printf 'true\\n'; fi\n exit 0\nfi\ncase \"${{1:-}}:${{2:-}}\" in\n network:create) : ;;\n network:inspect) printf '%s\\n' '{}' ;;\n network:rm) : ;;\n create:--name) printf '%s\\n' '{}' ;;\n start:*) : ;;\n container:inspect) printf '%s\\n' '{}' ;;\n top:*) printf '%s' '{}' ;;\n port:*) printf '\\377' ;;\n stop:*) : ;;\n rm:*) : ;;\n *) exit 91 ;;\nesac\n", - log.display(), - backend_info_json(), - network_inspection_json(), - FAKE_CONTAINER_ID, - container_inspection_json(), - process_security_output(), + "#!/bin/sh\nset -eu\nprintf '%s\\n' \"$*\" >> '{}'\nif [ \"${{1:-}}\" = info ] && [ \"${{3:-}}\" != json ]; then printf 'true\\n'; exit 0; fi\ncase \"${{1:-}}:${{2:-}}\" in\n info:--format) printf '%s\\n' '{SECURITY_INFO}' ;;\n network:create) : ;;\n network:inspect) printf '%s\\n' '{NETWORK_INSPECTION}' ;;\n create:--name) printf '%s\\n' '{FAKE_CONTAINER_ID}' ;;\n start:*) : ;;\n container:inspect) printf '%s\\n' '{CONTAINER_INSPECTION}' ;;\n top:*) printf 'PID SECCOMP CAPEFF CAPBND CAPINH CAPPRM CAPAMB LABEL\\n1 filter - - - - - containers-default (enforce)\\n' ;;\n port:*) printf '\\377' ;;\n stop:*) : ;;\n rm:*) : ;;\n network:rm) : ;;\n *) exit 91 ;;\nesac\n", + log.display() ); - let program = write_executable("non-utf8-podman", &script); + let program = write_executable(fixture.path(), "non-utf8-podman", &script); let adapter = RootlessPodmanAdapter::new(program.clone()); assert_eq!( @@ -146,32 +121,30 @@ fn non_utf8_port_output_fails_closed_and_cleans_every_created_resource() { #[test] fn cleanup_command_output_overflow_fails_closed_without_skipping_other_cleanup() { + let fixture = tempfile::tempdir().expect("isolated runtime boundary fixture directory"); let listener = TcpListener::bind(("127.0.0.1", 0)).expect("loopback listener should bind"); let ready_port = listener .local_addr() .expect("listener should expose its address") .port(); - let log = temporary_path("cleanup-output-limit-log"); + let log = fixture.path().join("cleanup-output-limit-log"); let script = format!( - "#!/bin/sh\nset -eu\nprintf '%s\\n' \"$*\" >> '{}'\nif [ \"${{1:-}}\" = info ]; then\n if [ \"${{3:-}}\" = json ]; then printf '%s\\n' '{}'; else printf 'true\\n'; fi\n exit 0\nfi\ncase \"${{1:-}}:${{2:-}}\" in\n network:create) : ;;\n network:inspect) printf '%s\\n' '{}' ;;\n network:rm) : ;;\n create:--name) printf '%s\\n' '{}' ;;\n start:*) : ;;\n container:inspect) printf '%s\\n' '{}' ;;\n top:*) printf '%s' '{}' ;;\n port:*) printf '127.0.0.1:{}\\n' ;;\n stop:*) i=0; while [ \"$i\" -lt 2048 ]; do printf x; i=$((i + 1)); done ;;\n rm:*) : ;;\n *) exit 91 ;;\nesac\n", + "#!/bin/sh\nset -eu\nprintf '%s\\n' \"$*\" >> '{}'\nif [ \"${{1:-}}\" = info ] && [ \"${{3:-}}\" != json ]; then printf 'true\\n'; exit 0; fi\ncase \"${{1:-}}:${{2:-}}\" in\n info:--format) printf '%s\\n' '{SECURITY_INFO}' ;;\n network:create) : ;;\n network:inspect) printf '%s\\n' '{NETWORK_INSPECTION}' ;;\n create:--name) printf '%s\\n' '{FAKE_CONTAINER_ID}' ;;\n start:*) : ;;\n container:inspect) printf '%s\\n' '{CONTAINER_INSPECTION}' ;;\n top:*) printf 'PID SECCOMP CAPEFF CAPBND CAPINH CAPPRM CAPAMB LABEL\\n1 filter - - - - - containers-default (enforce)\\n' ;;\n port:*) printf '127.0.0.1:{}\\n' ;;\n stop:*) i=0; while [ \"$i\" -lt 256 ]; do printf x; i=$((i + 1)); done ;;\n rm:*) : ;;\n network:rm) : ;;\n *) exit 91 ;;\nesac\n", log.display(), - backend_info_json(), - network_inspection_json(), - FAKE_CONTAINER_ID, - container_inspection_json(), - process_security_output(), - ready_port, + ready_port ); - let program = write_executable("cleanup-output-limit-podman", &script); - let adapter = RootlessPodmanAdapter::new(program.clone()) - .with_command_output_limit_bytes(1024) - .with_command_timeout(Duration::from_secs(1)); + let program = write_executable(fixture.path(), "cleanup-output-limit-podman", &script); + let adapter = + RootlessPodmanAdapter::new(program.clone()).with_command_timeout(Duration::from_secs(1)); let lease = adapter .launch_at(&request(&"a".repeat(64)), &policy(), 1_780_000_000) .expect("launch should succeed before bounded cleanup failure"); + let cleanup_adapter = RootlessPodmanAdapter::new(program.clone()) + .with_command_output_limit_bytes(64) + .with_command_timeout(Duration::from_secs(1)); assert_eq!( - adapter.terminate_at(&lease, 1_780_000_001), + cleanup_adapter.terminate_at(&lease, 1_780_000_001), Err(ApplicationServiceError::CleanupFailed) ); let calls = fs::read_to_string(&log).expect("all cleanup calls should be recorded");