You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Rate-limit both registrar verbs with two token buckets
Summary
Once a record is required for every registrar verb invocation, including refusals, the record store becomes the resource an attacker exhausts — and because the OpenBao file audit device on the same host is mandatory and OpenBao fails requests it cannot audit, a full disk stops far more than enrollment. This group bounds how fast one client identity may drive the two verbs. It is a grouping issue and carries no work of its own.
The seam between the two implementation children is mechanism versus evidence, and the third child writes no code at all. The limiter-mechanism child builds the limiter: two per-(client identity, verb) token buckets — a pre-derivation refusal bucket and a post-derivation admission bucket — both charged before the invocation's intent write, with a monotonic refill, a capped bucket map, four integer configuration keys and a written sizing formula, and a typed retryable throttle returned only where the outcome is genuinely undetermined. It suppresses records and emits an event; it writes nothing durable and puts nothing on the wire. The evidence child turns that suppressed traffic into evidence and tells the caller: it implements the sink as a coalescing window closed on the daemon's existing tick over a bounded map, writes one counted record per window per key as a third variant inside the audit store's existing types, amends both of the record reader's classification rules for it, supplies the match arm that produces the throttle's already-settled wire identifier, and publishes the limited-invocation counters as a health member.
They are strictly ordered — decision child, then limiter mechanism, then evidence — and the interface between the two implementation children is three things the limiter-mechanism child defines and exports: the two-value bucket enum, the limited-invocation sink trait, and the typed retryable outcome. That seam is deliberate — it lets the limiter be implemented, tested and reviewed before any record shape exists.
Children
Settle the limiter's bucket shape against the RFC — Detection (RFC amendment transcribing two rulings, no code)
Bound both registrar verbs with two token buckets — Detection
Record limited registrar invocations and report the throttle — Detection
Shared background
Suppression is both the point and the hazard. The limiter works by not writing the records a flood would otherwise produce — which is exactly what would make the flood invisible, since the mechanism that would have recorded it is the mechanism being skipped. The answer is one counted record per window per key rather than one per invocation: a sustained flood costs a bounded number of records while the fact of it, who drove it and which bucket ran dry all survive. Neither child may drift back toward per-invocation cost, and neither may drop the fact entirely.
Which bucket ran dry is carried all the way out, not summed on the way. The two buckets mean opposite things to whoever reads the signal: a predecision_refusal count rising says someone is flooding malformed input while those callers still received their real, permanent answers, and an admission count rising says legitimate traffic is being held back and a bring-up may be stalling. So the limiter-mechanism child's counter is two counters, the counted record names its bucket, and the health member publishes both rather than their sum — a total would put the less actionable number on the wire and leave the actionable one in a store on the bootroot host that no console can read.
Everything either child keys on the client identity must be bounded. The identity is opaque and may not be stable across connections — the limiter-mechanism child names the concrete case, a peer-credential value rendering the connecting pid, and raises it against the transport rather than repairing it here. Until it is stable, every per-identity structure is something a caller can multiply: the limiter-mechanism child caps its bucket map, the evidence child caps its map of open coalescing windows, and both spell out what happens at the cap rather than growing. An unbounded map in either child would be memory exhaustion on the bootroot host, delivered by the mechanism built to prevent exhaustion.
Both check points sit before the intent write, and that ordering is what forbids a whole class of key. At the moment the limiter can still prevent a durable write, the invocation's specific refusal reason does not exist yet — on the pre-derivation arm the checks have not run, and at admission the invocation might still turn out to be a perfectly good mint. So no bucket key and no coalescing key may carry the verb layer's fine-grained refusal reason, in either child. Beyond correctness, a key that varied with the refusal would let a caller multiply its budget simply by varying its input.
The fail-closed rule is not weakened anywhere in this group, and the distinction is worth stating precisely. "An invocation whose intent cannot be written is refused" is about a failed write. Suppression here is a deliberate skip, decided before any write is attempted, on paths that create nothing — and an accepted mint is never left unrecorded by it. Neither child may express suppression by adding a best-effort or skip mode to the record store: that store has none by design, and adding one would let suppression leak into the verb path the rule protects.
This group departed from the accepted RFC in two independent places. The RFC's owner has now ratified both, and the decision child exists to discharge them — the rulings are made, the document edit is not.
The bucket key — ratified. The RFC specifies per-(verb, outcome-class) buckets; this group ships per-(client identity, verb, bucket) buckets over a two-value enum, and that shape is the ruling. The reasoning is narrower than "the RFC's shape is not implementable as written", and the narrower statement is the one to carry: the two-value enum is the maximal refinement knowable at the point the limiter is charged. A pre-derivation refusal is knowable there — that arm fires only on refusals, and label validation and multiplicity-class resolution both precede the check point — while the outcome of an admitted invocation is not, because spec-conflict, name-collision and host-mismatch are settled by OpenBao work that follows the intent write. Charging on the outcome class would therefore require charging after the durable write the limiter exists to prevent, or predicting the outcome, and no third shape is specified today. State the delivered guarantee positively rather than only as a lost one: the cheapest flood path — malformed input and unknown components, refused before any OpenBao work — cannot starve accepted mints. That is exactly the path §5.6's own threat rationale names when it calls ServiceInstanceMismatch the cheapest refusal, and both of that error's causes are pre-derivation. What is no longer isolated is the expensive refusals: a caller able to produce well-formed, derivable requests consumes admission budget with invocations that end in refusal. That residual is bounded by attacker cost rather than by the limiter — every such attempt spends OpenBao work — and it is recorded with that reasoning attached rather than as a bare accepted weakening. RFC-F §5.6 states the per-(verb, outcome-class) shape as a guarantee, so ratifying this leaves a merged document asserting something the implementation will not deliver. That sentence must be amended to the delivered guarantee above — an obligation to discharge in the decision issue's RFC edit, not an implicit residual. Nothing in this group widens the bucket key toward outcome classes, and nothing re-opens the question: re-specifying it would mean designing a provisional-class-then-reconcile or post-outcome-shaping scheme first, and the change would reach the record store's serde encoding and golden fixture, the reader's pairing over coalesced records, and the cargo arm's bucket-isolation tests, all of which are written against the two-bucket shape today.
The caller answer — ratified. The RFC's rate-limit decision and its acceptance criterion both say a throttled invocation returns retryable RegistrarBusy without distinguishing check points; this group returns it from admission only, and that split is the ruling — an intentional correction to the RFC rather than a divergence to reconcile. The ground is type-level, not ergonomic.RegistrarBusy is the error family's only retryable member — the enroll wire issue exposes that as a classification method on the error type, true for RegistrarBusy and false for the other five — while the refusals the pre-derivation arm produces are classified deterministic and permanent by two other merged documents: RFC-D2 §4b terminates the attempt immediately on them and spends no retry budget, and RFC-E §9 states that ServiceInstanceMismatch is not retryable because clicking again sends the same request. Answering RegistrarBusy there would hand a caller the one value whose contract says this clears on its own for a request that can never succeed. It is operator-facing harm as well: RFC-E §9 requires the UI to render RegistrarBusy as an in-progress wait honouring retry_after, never as a failure, and to offer no retry button — so a malformed service_name would sit on screen as "waiting" indefinitely with its real cause unreachable. And it composes with the bucket ruling recorded beside it: the predecision_refusal bucket exists precisely because that arm is the cheapest, most floodable path, so answering RegistrarBusy on it would have the limiter feed the traffic it was added to damp. What the split must keep, and it is not optional. RFC-F §5.6 already says a throttled invocation on this arm "costs one coalesced counter increment rather than a durable record", so suppressing the two audit records is right — but the coalesced increment must still happen and the anomaly definition must still count it. Otherwise a flood becomes least visible exactly while it is worst, and these records are the only detection RFC 0001 §5.6's own "detected, not prevented" argument rests on. The two questions were independent — either bucket shape composes with either caller answer — and asking them separately is why this one was answered at all. Both are now settled, so the decision child's remaining job is the RFC amendment carrying both, which is what still gates the other two children: the mechanism child depends on it and the record child depends on the mechanism child.
One [registrar] table, two additive contributions to it, in a known order. The four bucket keys are the limiter-mechanism child's and the coalescing window key is the evidence child's; all are unsigned integers with documented defaults, all reject zero with an error naming the key, and all report a negative or non-integer value as the same named configuration error rather than a raw deserialization failure or a panic. Those two children are strictly ordered, so the bucket keys land first and the window key joins a table that already carries them: the later addition must leave every key already in the table untouched, including the audit-record work's audit_record_dir and rotation bounds and the reserved store's audit_store_* keys, which arrive from outside this group on no fixed schedule. The rule is "preserve what is there", not "either order works".
Neither child restructures the verbs. The two arms, the two locks, the request id, the opaque caller identity and the refusal taxonomy belong to the verbs issue and are consumed here. A check point that cannot be placed without reordering a check is a finding to raise there. The caller identity in particular is opaque to this group: if it is not stable across connections, the limiter is bypassable by hanging up and dialling again — but the repair belongs to the transport issue, not to a key this group parses or normalises.
"Limited" and "throttled" are different words in this group and are used precisely. An invocation is limited when the limiter suppressed its audit records — true at both check points, and the condition the counted record and the health counters describe. It is throttled when it is limited at the admission check point and therefore receives the retryable throttle outcome; a limited pre-derivation invocation is not throttled, because its refusal is already determined and telling such a caller to retry manufactures the flood the limiter exists to stop. Every throttled invocation is limited; the reverse does not hold. Neither child may use the two interchangeably, and no criterion about the throttle identifier may be phrased so that it appears to cover the pre-derivation case.
There is effectively one registrar identity, which is what makes both halves of this group necessary. "Per client identity" is therefore close to global: limiting an attacker limits the control plane in the same bucket. That is why the limiter is sized from the legitimate fan-out rather than from an intuition about steady state, and why an admission-throttled caller must receive a distinct, transient identifier — a throttle that arrives as a generic error gets classified as transient anyway, retried, and the retry storm feeds the limiter it was produced by.
Execution order
Issues in the same wave have no unmet dependencies among these children and can run in parallel.
graph TD
issue785["#785 Settle the limiter's bucket shape against the RFC [phase: Detection]"]
issue786["#786 Bound both registrar verbs with two token buckets [phase: Detection]"]
issue787["#787 Record limited registrar invocations and report the throttle [phase: Detection]"]
issue758["#758 Implement the restricted registrar mint and deregister verbs [phase: Verb building blocks]"]
issue758 -.-> issue786
issue777["#777 Write intent and outcome records around both registrar verb arms [phase: Detection]"]
issue777 -.-> issue786
issue785 --> issue786
issue785 --> issue787
issue786 --> issue787
issue776["#776 Add the append-only record store and its on-disk format for registrar audit records [phase: Detection]"]
issue776 -.-> issue787
issue779["#779 Scan the audit store for anomalies and report them on `bootroot status` [phase: Detection]"]
issue779 -.-> issue787
issue762["#762 Implement the registrar endpoint's versioned wire protocol and its codec [phase: Surface]"]
issue762 -.-> issue787
Rate-limit both registrar verbs with two token buckets
Summary
Once a record is required for every registrar verb invocation, including refusals, the record store becomes the resource an attacker exhausts — and because the OpenBao file audit device on the same host is mandatory and OpenBao fails requests it cannot audit, a full disk stops far more than enrollment. This group bounds how fast one client identity may drive the two verbs. It is a grouping issue and carries no work of its own.
The seam between the two implementation children is mechanism versus evidence, and the third child writes no code at all. The limiter-mechanism child builds the limiter: two per-
(client identity, verb)token buckets — a pre-derivation refusal bucket and a post-derivation admission bucket — both charged before the invocation's intent write, with a monotonic refill, a capped bucket map, four integer configuration keys and a written sizing formula, and a typed retryable throttle returned only where the outcome is genuinely undetermined. It suppresses records and emits an event; it writes nothing durable and puts nothing on the wire. The evidence child turns that suppressed traffic into evidence and tells the caller: it implements the sink as a coalescing window closed on the daemon's existing tick over a bounded map, writes one counted record per window per key as a third variant inside the audit store's existing types, amends both of the record reader's classification rules for it, supplies thematcharm that produces the throttle's already-settled wire identifier, and publishes the limited-invocation counters as a health member.They are strictly ordered — decision child, then limiter mechanism, then evidence — and the interface between the two implementation children is three things the limiter-mechanism child defines and exports: the two-value bucket enum, the limited-invocation sink trait, and the typed retryable outcome. That seam is deliberate — it lets the limiter be implemented, tested and reviewed before any record shape exists.
Children
Shared background
Suppression is both the point and the hazard. The limiter works by not writing the records a flood would otherwise produce — which is exactly what would make the flood invisible, since the mechanism that would have recorded it is the mechanism being skipped. The answer is one counted record per window per key rather than one per invocation: a sustained flood costs a bounded number of records while the fact of it, who drove it and which bucket ran dry all survive. Neither child may drift back toward per-invocation cost, and neither may drop the fact entirely.
Which bucket ran dry is carried all the way out, not summed on the way. The two buckets mean opposite things to whoever reads the signal: a
predecision_refusalcount rising says someone is flooding malformed input while those callers still received their real, permanent answers, and anadmissioncount rising says legitimate traffic is being held back and a bring-up may be stalling. So the limiter-mechanism child's counter is two counters, the counted record names its bucket, and the health member publishes both rather than their sum — a total would put the less actionable number on the wire and leave the actionable one in a store on the bootroot host that no console can read.Everything either child keys on the client identity must be bounded. The identity is opaque and may not be stable across connections — the limiter-mechanism child names the concrete case, a peer-credential value rendering the connecting pid, and raises it against the transport rather than repairing it here. Until it is stable, every per-identity structure is something a caller can multiply: the limiter-mechanism child caps its bucket map, the evidence child caps its map of open coalescing windows, and both spell out what happens at the cap rather than growing. An unbounded map in either child would be memory exhaustion on the bootroot host, delivered by the mechanism built to prevent exhaustion.
Both check points sit before the intent write, and that ordering is what forbids a whole class of key. At the moment the limiter can still prevent a durable write, the invocation's specific refusal reason does not exist yet — on the pre-derivation arm the checks have not run, and at admission the invocation might still turn out to be a perfectly good mint. So no bucket key and no coalescing key may carry the verb layer's fine-grained refusal reason, in either child. Beyond correctness, a key that varied with the refusal would let a caller multiply its budget simply by varying its input.
The fail-closed rule is not weakened anywhere in this group, and the distinction is worth stating precisely. "An invocation whose intent cannot be written is refused" is about a failed write. Suppression here is a deliberate skip, decided before any write is attempted, on paths that create nothing — and an accepted mint is never left unrecorded by it. Neither child may express suppression by adding a best-effort or skip mode to the record store: that store has none by design, and adding one would let suppression leak into the verb path the rule protects.
This group departed from the accepted RFC in two independent places. The RFC's owner has now ratified both, and the decision child exists to discharge them — the rulings are made, the document edit is not.
The bucket key — ratified. The RFC specifies per-
(verb, outcome-class)buckets; this group ships per-(client identity, verb, bucket)buckets over a two-value enum, and that shape is the ruling. The reasoning is narrower than "the RFC's shape is not implementable as written", and the narrower statement is the one to carry: the two-value enum is the maximal refinement knowable at the point the limiter is charged. A pre-derivation refusal is knowable there — that arm fires only on refusals, and label validation and multiplicity-class resolution both precede the check point — while the outcome of an admitted invocation is not, because spec-conflict, name-collision and host-mismatch are settled by OpenBao work that follows the intent write. Charging on the outcome class would therefore require charging after the durable write the limiter exists to prevent, or predicting the outcome, and no third shape is specified today. State the delivered guarantee positively rather than only as a lost one: the cheapest flood path — malformed input and unknown components, refused before any OpenBao work — cannot starve accepted mints. That is exactly the path §5.6's own threat rationale names when it callsServiceInstanceMismatchthe cheapest refusal, and both of that error's causes are pre-derivation. What is no longer isolated is the expensive refusals: a caller able to produce well-formed, derivable requests consumes admission budget with invocations that end in refusal. That residual is bounded by attacker cost rather than by the limiter — every such attempt spends OpenBao work — and it is recorded with that reasoning attached rather than as a bare accepted weakening. RFC-F §5.6 states the per-(verb, outcome-class)shape as a guarantee, so ratifying this leaves a merged document asserting something the implementation will not deliver. That sentence must be amended to the delivered guarantee above — an obligation to discharge in the decision issue's RFC edit, not an implicit residual. Nothing in this group widens the bucket key toward outcome classes, and nothing re-opens the question: re-specifying it would mean designing a provisional-class-then-reconcile or post-outcome-shaping scheme first, and the change would reach the record store'sserdeencoding and golden fixture, the reader's pairing over coalesced records, and the cargo arm's bucket-isolation tests, all of which are written against the two-bucket shape today.The caller answer — ratified. The RFC's rate-limit decision and its acceptance criterion both say a throttled invocation returns retryable
RegistrarBusywithout distinguishing check points; this group returns it from admission only, and that split is the ruling — an intentional correction to the RFC rather than a divergence to reconcile. The ground is type-level, not ergonomic.RegistrarBusyis the error family's only retryable member — the enroll wire issue exposes that as a classification method on the error type, true forRegistrarBusyand false for the other five — while the refusals the pre-derivation arm produces are classified deterministic and permanent by two other merged documents: RFC-D2 §4b terminates the attempt immediately on them and spends no retry budget, and RFC-E §9 states thatServiceInstanceMismatchis not retryable because clicking again sends the same request. AnsweringRegistrarBusythere would hand a caller the one value whose contract says this clears on its own for a request that can never succeed. It is operator-facing harm as well: RFC-E §9 requires the UI to renderRegistrarBusyas an in-progress wait honouringretry_after, never as a failure, and to offer no retry button — so a malformedservice_namewould sit on screen as "waiting" indefinitely with its real cause unreachable. And it composes with the bucket ruling recorded beside it: thepredecision_refusalbucket exists precisely because that arm is the cheapest, most floodable path, so answeringRegistrarBusyon it would have the limiter feed the traffic it was added to damp. What the split must keep, and it is not optional. RFC-F §5.6 already says a throttled invocation on this arm "costs one coalesced counter increment rather than a durable record", so suppressing the two audit records is right — but the coalesced increment must still happen and the anomaly definition must still count it. Otherwise a flood becomes least visible exactly while it is worst, and these records are the only detection RFC 0001 §5.6's own "detected, not prevented" argument rests on. The two questions were independent — either bucket shape composes with either caller answer — and asking them separately is why this one was answered at all. Both are now settled, so the decision child's remaining job is the RFC amendment carrying both, which is what still gates the other two children: the mechanism child depends on it and the record child depends on the mechanism child.One
[registrar]table, two additive contributions to it, in a known order. The four bucket keys are the limiter-mechanism child's and the coalescing window key is the evidence child's; all are unsigned integers with documented defaults, all reject zero with an error naming the key, and all report a negative or non-integer value as the same named configuration error rather than a raw deserialization failure or a panic. Those two children are strictly ordered, so the bucket keys land first and the window key joins a table that already carries them: the later addition must leave every key already in the table untouched, including the audit-record work'saudit_record_dirand rotation bounds and the reserved store'saudit_store_*keys, which arrive from outside this group on no fixed schedule. The rule is "preserve what is there", not "either order works".Neither child restructures the verbs. The two arms, the two locks, the request id, the opaque caller identity and the refusal taxonomy belong to the verbs issue and are consumed here. A check point that cannot be placed without reordering a check is a finding to raise there. The caller identity in particular is opaque to this group: if it is not stable across connections, the limiter is bypassable by hanging up and dialling again — but the repair belongs to the transport issue, not to a key this group parses or normalises.
"Limited" and "throttled" are different words in this group and are used precisely. An invocation is limited when the limiter suppressed its audit records — true at both check points, and the condition the counted record and the health counters describe. It is throttled when it is limited at the admission check point and therefore receives the retryable throttle outcome; a limited pre-derivation invocation is not throttled, because its refusal is already determined and telling such a caller to retry manufactures the flood the limiter exists to stop. Every throttled invocation is limited; the reverse does not hold. Neither child may use the two interchangeably, and no criterion about the throttle identifier may be phrased so that it appears to cover the pre-derivation case.
There is effectively one registrar identity, which is what makes both halves of this group necessary. "Per client identity" is therefore close to global: limiting an attacker limits the control plane in the same bucket. That is why the limiter is sized from the legitimate fan-out rather than from an intuition about steady state, and why an admission-throttled caller must receive a distinct, transient identifier — a throttle that arrives as a generic error gets classified as transient anyway, retried, and the retry storm feeds the limiter it was produced by.
Execution order
Issues in the same wave have no unmet dependencies among these children and can run in parallel.
External dependencies:
bootroot status#779, Implement the registrar endpoint's versioned wire protocol and its codec #762