Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -549,6 +549,21 @@ service (`managed_retention/pressure.rs`); the planner only gets the policy. A
root the daemon cannot stat is no pressure, logged once. The pass logs
`bosn retention: disk pressure (X GiB free, Y%); idle volume budget 0`.

## CI late-create cleanup (#676)

An unacknowledged Docker create keeps its durable helper or engine intent while
the request may still land. The retry loop accepts a fresh, explicit Docker
absence observation only after the 30 s control deadline plus 10 min grace
(630 s). Helper intents use their immutable creation time; engine intents use
the last durable update, which can only extend the wait. The engine records the
create request at the moment it is sent, after preparation. Startup migrates a
legacy engine without a create marker once into a durable uncertain marker
with a fresh timestamp; later retries use that timestamp. An absent helper is
finished without inventing a container ID. Any observed object still goes
through exact ownership and removal checks. Repeated deferrals log once per
intent and reason until that intent retires; the log state is process-local and
does not affect cleanup decisions.

## Shared CI cache budget (#544)

Decided 2026-10-10, moved behind act2 the same day (#547 step 4). While no
Expand Down
18 changes: 18 additions & 0 deletions crates/bosn-registry/src/act.rs
Original file line number Diff line number Diff line change
Expand Up @@ -582,6 +582,24 @@ impl Immediate<'_> {
record.updated_at = record.updated_at.max(at);
self.store_act_record(&record)
}
/// Historical records have no create-request marker. Start their settle
/// window once at recovery, then preserve that timestamp across retries.
pub fn mark_legacy_act_create_uncertain(&mut self, run: &str, at: f64) -> Result<(), Error> {
let mut record = self
.act_record(run)?
.ok_or(Error::BadRow("act intent missing"))?;
if record.state != ActEngineState::CleanupRequired
|| record.engine_id.is_some()
|| record.create_requested.is_some()
|| !at.is_finite()
|| at < record.updated_at
{
return Err(Error::BadRow("legacy act create uncertainty"));
}
record.create_requested = Some(true);
record.updated_at = at;
self.store_act_record(&record)
}
/// Exact ownership predicate for both registration and pending-intent
/// crash recovery. A deterministic name alone never authorizes adoption.
pub fn verify_act_engine(
Expand Down
26 changes: 25 additions & 1 deletion crates/bosn-registry/src/cache_helper.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ use serde::{Deserialize, Serialize};
const PREFIX: &str = "ci.cache-helper.v1:";
const RECORD_BYTES: usize = 4096;
pub const MAX_HELPER_PAGE: usize = 64;
/// Docker control deadline (30 s) plus ten minutes for a late create to settle.
pub const LATE_CREATE_WINDOW_SECS: f64 = 630.0;

#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
Expand Down Expand Up @@ -101,7 +103,11 @@ fn decode(detail: &str, nonce: &str, owner: &str) -> Result<CacheHelperRecord, E
|| record.intent.nonce != nonce
|| !record.updated_at.is_finite()
|| record.updated_at < record.intent.created_at
|| (record.state == CacheHelperState::Pending) != record.container_id.is_none()
|| match record.state {
CacheHelperState::Pending => record.container_id.is_some(),
CacheHelperState::Created => record.container_id.is_none(),
CacheHelperState::Removed => false,
}
|| record.container_id.as_ref().is_some_and(|id| !hex_id(id))
{
return Err(Error::BadRow("cache helper snapshot"));
Expand Down Expand Up @@ -217,6 +223,24 @@ impl Immediate<'_> {
record.updated_at = at;
self.store_helper(&record)
}

/// A trusted Docker name probe may finish an unacknowledged create only
/// after its bounded late-create window. No immutable ID is fabricated.
pub fn finish_absent_cache_helper(&mut self, nonce: &str, at: f64) -> Result<(), Error> {
let mut record = self
.helper_record(nonce)?
.ok_or(Error::BadRow("cache helper intent missing"))?;
if record.state != CacheHelperState::Pending
|| record.container_id.is_some()
|| !at.is_finite()
|| at < record.intent.created_at + LATE_CREATE_WINDOW_SECS
{
return Err(Error::BadRow("cache helper late-create absence proof"));
}
record.state = CacheHelperState::Removed;
record.updated_at = at;
self.store_helper(&record)
}
}

impl Registry {
Expand Down
29 changes: 29 additions & 0 deletions crates/bosn-registry/src/cache_helper/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,35 @@ fn pending_create_survives_restart_and_cannot_finish_without_a_known_id() {
);
}

#[test]
fn aged_absence_receipt_finishes_without_fabricating_an_id() {
let dir = fs::TemporaryDirectory::new().unwrap();
let path = dir.path().join("registry.sqlite3");
let value = intent(141);
let mut registry = Registry::create_writer(&path, OWNER).unwrap();
let mut tx = registry.begin_immediate().unwrap();
tx.begin_cache_helper(&value).unwrap();
assert!(
tx.finish_absent_cache_helper(&value.nonce, LATE_CREATE_WINDOW_SECS)
.is_err()
);
tx.finish_absent_cache_helper(&value.nonce, 1.0 + LATE_CREATE_WINDOW_SECS)
.unwrap();
tx.commit().unwrap();
drop(registry);
let reader = Registry::open_writer(&path).unwrap();
let record = reader.cache_helper(&value.nonce).unwrap().unwrap();
assert_eq!(record.state, CacheHelperState::Removed);
assert!(record.container_id.is_none());
assert!(
reader
.pending_cache_helpers(None, 64)
.unwrap()
.items
.is_empty()
);
}

#[test]
fn helper_identity_conflicts_and_invalid_receipts_leave_the_claim_intact() {
let dir = fs::TemporaryDirectory::new().unwrap();
Expand Down
3 changes: 2 additions & 1 deletion crates/bosn-service/src/act_engine/create.rs
Original file line number Diff line number Diff line change
Expand Up @@ -469,12 +469,13 @@ pub(super) async fn create_owned_engine_inner(
}
ensure_storage_volume(engine, &intent, owner).await?;
socket::of(&intent).map_or(Ok(()), socket::ensure_dir)?;
let at = crate::act_runtime::now().max(at).max(intent.created_at);
// From here an engine may exist even if this daemon never learns its ID, so its absence can
// no longer be proven by an empty lookup. Before here it can (#554).
registry
.act_registry(ActRegistryCommand::CreateRequested {
run: intent.run_id.clone(),
at: at.max(intent.created_at),
at,
})
.await
.map_err(|e| ActEngineError(e.to_string()))?;
Expand Down
11 changes: 6 additions & 5 deletions crates/bosn-service/src/act_engine/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -593,14 +593,15 @@ else: sys.exit(9)
let (sender, receiver) = async_engine::channel(16);
let actor = RegistryActor { sender };
let task = async_engine::launch(crate::registry_actor(writer, receiver, None));
let at = crate::act_runtime::now();
let created = create_owned_engine(
&actor,
&engine,
intent(),
OWNER,
&format!("sha256:{}", "2".repeat(64)),
limits(),
1.0,
at,
)
.await;
if matches!(mode, "foreign" | "start-failed") {
Expand All @@ -613,7 +614,7 @@ else: sys.exit(9)
intent: intent(),
observed: observed.clone(),
token: token.into(),
at: 2.0,
at: at + 1.0,
})
.await
.unwrap();
Expand All @@ -622,7 +623,7 @@ else: sys.exit(9)
run: intent().run_id,
token: token.into(),
outcome: ActRunOutcome::Passed,
at: 2.0,
at: at + 1.0,
})
.await
.unwrap();
Expand All @@ -631,12 +632,12 @@ else: sys.exit(9)
run: intent().run_id,
token: token.into(),
outcome: ActRunOutcome::Passed,
at: 2.0,
at: at + 1.0,
})
.await
.unwrap();
assert_eq!(
remove_owned_engine(&actor, &engine, &intent().run_id, observed, 2.0)
remove_owned_engine(&actor, &engine, &intent().run_id, observed, at + 1.0)
.await
.is_ok(),
matches!(mode, "success" | "slow-remove")
Expand Down
16 changes: 16 additions & 0 deletions crates/bosn-service/src/act_registry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,10 @@ pub enum ActRegistryCommand {
id: String,
at: f64,
},
HelperAbsent {
nonce: String,
at: f64,
},
HelperPending {
after_nonce: Option<String>,
limit: usize,
Expand Down Expand Up @@ -204,6 +208,11 @@ pub(crate) fn apply(
return Err(bosn_registry::Error::BadRow("act startup terminal intent"));
}
if record.state == bosn_registry::act::ActEngineState::CleanupRequired {
if record.engine_id.is_none() && record.create_requested.is_none() {
let mut transaction = registry.begin_immediate()?;
transaction.mark_legacy_act_create_uncertain(run, *at)?;
transaction.commit()?;
}
return Ok(ActRegistryReply::Committed);
}
let mut transaction = registry.begin_immediate()?;
Expand All @@ -217,6 +226,9 @@ pub(crate) fn apply(
} else {
transaction.request_act_cleanup(run, ActRunOutcome::Interrupted, *at)?;
}
if record.engine_id.is_none() && record.create_requested.is_none() {
transaction.mark_legacy_act_create_uncertain(run, *at)?;
}
transaction.commit()?;
return Ok(ActRegistryReply::Committed);
}
Expand Down Expand Up @@ -365,6 +377,10 @@ pub(crate) fn apply(
transaction.finish_cache_helper(&nonce, &id, at)?;
ActRegistryReply::Committed
}
ActRegistryCommand::HelperAbsent { nonce, at } => {
transaction.finish_absent_cache_helper(&nonce, at)?;
ActRegistryReply::Committed
}
ActRegistryCommand::Pending { .. }
| ActRegistryCommand::MaintenanceLatest
| ActRegistryCommand::Get { .. }
Expand Down
25 changes: 20 additions & 5 deletions crates/bosn-service/src/act_runtime/startup.rs
Original file line number Diff line number Diff line change
Expand Up @@ -91,13 +91,22 @@ pub async fn recover_startup_act_engines(
})
.await
.map_err(|e| error(e.to_string()))?;
let reply = registry
.act_registry(ActRegistryCommand::Get {
run: record.intent.run_id.clone(),
})
.await
.map_err(|e| error(e.to_string()))?;
let ActRegistryReply::Record(Some(updated)) = reply else {
return Err(error("startup interrupted record is missing"));
};
let remaining = budget.saturating_sub(started.elapsed());
let result = retire_engine(
registry,
engine,
owner,
proofs,
&record,
&updated,
remaining,
cancellation,
)
Expand Down Expand Up @@ -327,11 +336,17 @@ pub(crate) async fn retire_engine(
.map_err(|e| error(format!("engine removal: {e}")))
}

/// An empty lookup proves removal only after this creation was observed, or when `docker
/// create` was provably never sent (#554). No waiting period establishes that an unresolved
/// Docker request is done.
/// An empty lookup closes an unresolved create only after the last durable
/// record update has aged past the Docker control deadline plus grace. Using
/// `updated_at` conservatively restarts the wait after a later interruption.
pub(crate) fn confirm_absence(record: &bosn_registry::act::ActEngineRecord) -> std::io::Result<()> {
if record.engine_id.is_none() && record.create_requested != Some(false) {
if record.engine_id.is_none() && record.create_requested.is_none() {
return Err(error("legacy engine create window has not been started"));
}
if record.engine_id.is_none()
&& record.create_requested == Some(true)
&& now() - record.updated_at < bosn_registry::cache_helper::LATE_CREATE_WINDOW_SECS
{
return Err(error(
"engine creation remains unresolved; retaining cleanup_required for reconciliation",
));
Expand Down
Loading
Loading