You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Status — both questions settled; blocked only on the RFC amendment
Do not implement it as filed yet, but nothing in it is unsettled. The specification below departed from the accepted RFC in two independent places, and the RFC's owner has now ratified both exactly as specified here: the bucket key as per-(client identity, verb, bucket) buckets over the two-value enum, and the caller answer as the throttle from the admission check point only, with a limited pre-derivation invocation keeping its real permanent refusal. Everything from the Context down is authorised. What remains is bookkeeping with teeth — the sibling decision issue must land the RFC amendment carrying both rulings, because until it does the accepted document still states guarantees this code will not deliver.
The bucket key — settled, ratified as written. The RFC (§5.6) specifies per-(verb, outcome-class) buckets "so refusals throttle without starving accepted mints"; this issue's two-value shape stands, and Departure 1 below now records the ratified reasoning rather than a proposal. Nothing in the Scope changes.
What a limited pre-derivation invocation tells its caller — settled, ratified as written. The RFC (§5.6, restated in §6 and §7) says a throttled invocation returns retryable RegistrarBusy { retry_after }, and its pre-derivation-arm decision calls a records-suppressed invocation on that arm "throttled" — so the RFC as written implies that answer on both check points. This issue returns the throttle from the admission check point only, and that split is the ruling. Departure 2 below records the ratified reasoning. Nothing in the Scope changes.
The unblocking procedure. This section is the whole of it — no other artifact needs to be consulted to know what happens when the amendment lands.
What unblocks this issue — stated once here, and referenced rather than restated everywhere else in this issue: the sibling decision issue lands its RFC amendment carrying both rulings. Both are decided; neither is discharged until the edit lands. §5.6 states the per-(verb, outcome-class) bucket shape as a guarantee, and §5.6 and §6 both state the RegistrarBusy answer without qualifying the check point — so until the amendment lands the accepted document asserts two things this code will not deliver. Neither half alone unblocks this issue, and rulings recorded only in the decision issue's discussion, without the RFC edit, do not either.
The editorial step to apply at that moment, and it is purely editorial: delete this Status section; fold Departure 1's charge-point reasoning into the Scope bullet that defines the bucket key, where it survives as design rationale; then delete the whole Why this specification departs from the accepted RFC section, both subsections with it, since once the RFC carries the amendment neither one describes a departure any longer. Both rulings match what the Scope already says, so no spec text needs rewriting and nothing else in the Scope moves.
No substance in the Scope changes when the amendment lands — the editorial step above is the entire edit.
Both questions are closed and neither reopens. Ratified and not to be revisited here: the bucket key, the two-value bucket enum exported to the sibling record issue, the bounded-map and overflow-bucket rules stated in terms of that key, the two pairs of rate_limit_* keys, the isolation guarantee below, the admission-only throttle, and a limited pre-derivation invocation keeping its real permanent refusal while its records are suppressed. Do not widen the bucket key toward outcome classes, do not widen the throttle to the pre-derivation arm, and do not treat a later reading of the unamended RFC sentences as reopening either — those sentences are the amendment's to fix.
Context
A record is now required for every registrar verb invocation, including refusals. That closes a detection gap and opens a resource one: there is no limit anywhere on either verb, so a compromised caller — the exact actor this whole mechanism exists for — can flood refused invocations and grow the record store without bound. The cheapest refusals are the ones rejected before any OpenBao work, so the flood path is also the free path.
The consequence is not merely a large file. When the filesystem fills, every subsequent mint is refused, so fleet-wide enrollment stops. Worse, the OpenBao file audit device this design builds on is mandatory — openbao/openbao.hcl declares an audit { type = "file" ... } stanza and bootroot init refuses to proceed without it (src/openbao.rs::verify_audit_file) — and OpenBao fails requests when it cannot write its audit device. So a full disk can stop OpenBao serving, which is the renewal path for every certificate in the deployment, on the one host that must not be restarted. "Mints are rare in steady state" is true of legitimate mints and irrelevant to the attacker.
This issue builds the mechanism that bounds the rate: the buckets, where they are charged, and what each check point tells its caller. Two words are used precisely throughout and are not synonyms: an invocation is limited when this issue suppressed its audit records, which happens at both check points, and throttled when it is limited at the admission check point and therefore receives the retryable throttle outcome. Every throttled invocation is limited; the reverse does not hold. A sibling issue turns the invocations this one suppresses into counted records, amends the record reader for them, and publishes the counter on the endpoint. A third, unrelated issue bounds the space those records land in. A per-(component, host)ceiling on how many instances a deployment may legitimately run is a fourth object again, and is deliberately deferred.
Scope
Two buckets, both consulted before any durable write
The limiter has exactly two check points, and each sits before the invocation's intent record is written. This ordering is forced, not stylistic: the audit-record work writes the intent at arm entry — on the pre-derivation arm, before that arm's checks run — so at every point where the limiter can still prevent a durable write, the specific refusal reason is not yet known. The buckets are therefore keyed on where the invocation is, not on what it will turn out to be.
Bucket key: (client identity, verb, bucket), where bucket is an enum with exactly two values, defined here and exported for the sibling record issue to reuse:
predecision_refusal — the verb layer's pre-derivation arm: the arm entered for a service_name or host that is not a DNS label, or a component with no multiplicity entry. Every invocation on this arm is a refusal by construction, it performs no OpenBao work, and it is the cheapest thing a caller can make the daemon do.
admission — every invocation that reaches the post-derivation arm and may therefore reach OpenBao, whatever it turns out to be.
Two verbs × two buckets gives four buckets per client identity.
The bucket key never contains the verb layer's fine-grained refusal reason. It is unknown at both check points, and a key that varied with it would let a caller multiply its budget simply by varying its refusal.
The client identity is the opaque value the verb layer already carries into every invocation. Key the buckets on it as-is. It is coarse before mTLS lands (the socket's peer credentials) and becomes the client certificate's identity afterwards, with no change here and no dependency on the mTLS work to obtain it.
That value must be stable for one caller across connections, and this issue asserts it rather than assuming it. A bucket is created full and keyed on the identity, so an identity that varies per connection gives every reconnect a fresh full budget: the limiter is then bypassable by hanging up and dialling again, which is free for the caller and defeats the control entirely. It also makes the bucket map grow without bound, on the daemon this issue exists to protect. The hazard is concrete rather than hypothetical — a peer-credential value that renders the connecting pid varies on every connection by construction. So write the stability test below; if the transport's value does not have that property, that is a finding to raise against the transport issue, not something to repair here by parsing, normalising or stripping fields out of a value this issue is forbidden to interpret.
Bound the map with an explicit cap, counted in bucket entries. The limiter holds one entry per (client identity, verb, bucket) key, and the cap is a file-top constant on the number of entries, not on the number of identities: 256, which is the reference deployment's 64 identities × the four buckets each identity can create. Entries are what consume the memory, and counting them keeps the cap in the same unit as every removal rule below — a cap counted in identities cannot be honoured by removing a bucket at all, since an identity holding one drained bucket and three full ones is not removable and freeing its three full ones frees no identity slot. There is effectively one registrar identity, so the cap is never reached in normal operation; it exists because the identity-stability failure above must not become memory exhaustion on the bootroot host.
What a charge does, stated exhaustively, because this path is the whole defence against a caller-varying identity. A charge whose key is already in the map charges that entry. A charge whose key is absent:
Below the cap — insert a bucket holding a full burst, and charge it.
At the cap — first reclaim every entry currently refilled to its full burst. A full entry is indistinguishable from an absent one, since both yield a full bucket on next use, so this frees slots without handing anybody a token. It never touches a drained entry, and it needs no victim-selection policy: the rule is every full entry or none, which is deterministic and directly testable. If it frees a slot, insert and charge as above.
Still at the cap — every remaining entry is drained, so insert nothing and charge the invocation against the overflow bucket for its (verb, bucket).
The four overflow buckets. One per (verb, bucket) pair, created with the limiter, sized from the same configured burst and refill interval as the map entries for that bucket, never reclaimed, and not counted against the cap. They exist to make the miss path total: whatever the map holds, there is always exactly one bucket to charge, so no invocation is ever admitted unlimited for want of a slot. Charging "some existing entry for the same (verb, bucket)" would not be total — a saturated map need not contain that pair at all, which is reachable whenever traffic is mixed and buckets are created lazily — and it would spend one identity's budget on another's traffic by an unspecified choice of victim.
The residual this leaves, stated rather than implied. While the map is saturated with drained entries, every identity without an entry of its own shares the four overflow buckets, so in that degraded state one caller's flood can throttle another caller's traffic. That is inherent to any bounded map — memory cannot be bounded while a private bucket is guaranteed to an unbounded number of identities — and what matters is that it is bounded rather than unlimited. It is also unreachable in normal operation: getting there needs the identity-stability defect above, whose fix is the finding raised against the transport, not a larger cap here.
The limiter is a token bucket, not a fixed or sliding window. A fixed window lets a caller spend two full budgets back to back across a boundary; a token bucket states the legitimate burst and the sustained rate as two independent numbers, which is exactly the distinction the sizing below turns on. Tokens accrue at one token per refill interval, are capped at the burst, and are computed from a monotonic clock — a wall-clock step must not be able to mint tokens.
A bucket starts full, is held in memory only, and is created lazily. A bucket comes into existence on its key's first use holding a full burst, so a freshly started endpoint can absorb the full documented bring-up wave immediately — which is exactly what the sizing formula promises an operator, and an empty-start bucket would refuse the first wave on every restart. Nothing about the limiter is persisted across restarts: durable limiter state would have to live on the very store the reserved-store work is protecting, and it would have to survive the crash cases the audit-record work already treats as anomalies. The cost is that restarting the daemon resets every bucket — acceptable, because restarting the daemon requires root on the bootroot host, and root compromise of that host is outside what this design defends against.
An entry is removed only while it holds a full burst — never below it, anywhere, for any reason. That is the rule the reclaim step above obeys, and it governs any other place the map is pruned. Removing a full entry is indistinguishable from recreating it; removing a drained one silently hands that caller a fresh budget, which is a limiter bypass reachable by waiting. Reclaiming full entries at other times — an idle sweep, say — is permitted and changes no observable behaviour; reclaiming them at the cap is required, because it is what lets the map drain back down after a burst of identity churn instead of staying saturated until the daemon restarts.
Check point 1 — pre-derivation arm entry, before the intent write. Consult and charge the predecision_refusal bucket at the moment the arm is entered and before its intent record is written. With a token available, the arm proceeds exactly as the audit-record work specifies: intent, checks, outcome, paired records. With the bucket empty, skip both record writes, emit one limited-invocation event (below), and otherwise let the arm run unchanged — its checks are pure input validation with no I/O — so the caller receives its real, permanent refusal, not a throttle. This is the free-flood path, and it is the only place where records are suppressed and the caller is still given its real answer.
Check point 2 — post-derivation arm entry, before the intent write and before any OpenBao work. Consult and charge the admission bucket. With the bucket empty the invocation is not attempted at all: it returns the typed throttle outcome below, writes no intent and no outcome, and emits one limited-invocation event.
The throttle outcome
The throttle is returned only from check point 2, and that is settled by ruling rather than preference. It means the outcome is undetermined and a later attempt may succeed — true at admission, false on the pre-derivation arm, where the request is invalid on its face and always will be. Retryability is a property of the error type, not a tone: RegistrarBusy is the family's only retryable member, and this arm's refusals are classified permanent by the merged documents the sibling record issue maps onto. Reporting one as retryable would also make a broken caller retry a request that can never succeed, and that retry storm is itself a flood. So throttling never converts a determined outcome into a retryable one; on the pre-derivation arm the limiter suppresses the record, never the answer.
Suppressing the record is not suppressing the evidence — and this issue's entire share of that is emitting the event. A limited pre-derivation invocation still emits its one limited-invocation event through the sink, exactly as check point 1 requires, and that emission is the whole obligation here. Coalescing those events into a counted record, and amending the record reader's anomaly definition to account for them, belong to the sibling record issue's sink and are listed in Out of scope; do not implement or update a coalescing counter in this issue. RFC-F §5.6 describes the end state that chain produces — such an invocation "costs one coalesced counter increment rather than a durable record" — which is why the event may never be skipped along with the records: skip it and the chain has nothing to count, making a flood least visible exactly while it is worst, and these records are the only detection the RFC's "detected, not prevented" argument rests on.
Produce it as a distinct variant in the verb layer's own typed outcome surface, classified retryable, carrying a retry-after duration. It must be distinguishable in that surface from every permanent unavailability variant, whose reasons all mean until an operator acts, so that the sibling record issue can map it onto its own wire identifier without inspecting a boolean or a string.
The payload is a duration, not a deadline: an unsigned whole-second count, never an absolute timestamp, so it does not depend on the caller's clock agreeing with the daemon's. Its value is the time until the admission bucket holds one token, derived from the bucket's elapsed accrual against the admission refill interval and rounded up to whole seconds, with a floor of one second, so a value of 0 can never invite an immediate retry. It is deterministic; document that a caller with several outstanding requests should jitter its own retries.
What the two buckets do and do not guarantee — stated plainly, because the boundary matters. A flood on the predecision_refusal path cannot consume admission budget, so the path an attacker can drive for free, with no valid input and no OpenBao cost, cannot starve legitimate mints. That is the isolation this design buys, and it is the flood the Context describes. A caller that can produce well-formed, derivable requests can still consume admission budget with invocations that end in refusal, because at admission nothing distinguishes them from a real mint. That is bounded by attacker cost rather than by the limiter — every such attempt spends OpenBao work as well as an admission token — but it is not isolated from legitimate mints, and the documentation must say so rather than imply a guarantee the design does not provide. State the delivered guarantee positively while doing so: the isolation above is exactly the path §5.6's own threat rationale names when it calls ServiceInstanceMismatch the cheapest refusal, and both of that error's causes are pre-derivation. Bounding the expensive path more tightly would require classifying the outcome before the intent write, which the record ordering forbids.
Why this specification departs from the accepted RFC
Two departures, not one, and the second is not a consequence of the first — either bucket shape composes with either caller answer. Both have been ruled on, and each is recorded below as the settled reasoning the RFC amendment carries. Neither is an open question, and neither may be reopened, renegotiated or closed differently inside this issue. This entire section is temporary. The Status section's editorial step deletes it — both subsections — once the amendment lands, after folding Departure 1's charge-point reasoning into the Scope bullet that defines the bucket key, so that reasoning is kept while the departure framing goes.
Departure 1 — the bucket key. Ratified; recorded here as the reasoning the RFC amendment carries. The RFC specifies per-(verb, outcome-class) buckets "so refusals throttle without starving accepted mints" — a stronger guarantee than the two buckets above deliver, since it would isolate every class of refusal from legitimate mints rather than only the pre-derivation class. The accurate reason is not that the RFC's shape is unimplementable, and this issue does not claim that. It is that the two-value enum is the maximal refinement knowable at the point the limiter is charged: a pre-derivation refusal is knowable there, because that arm fires only on refusals and label validation and multiplicity-class resolution both precede the check point, while the outcome of an admitted invocation is not, because spec-conflict, name-collision and host-mismatch are settled by OpenBao work that follows the intent write. Keying on the outcome class would therefore require charging after the durable write the limiter exists to prevent, or predicting the outcome, and no third shape is specified today — a provisional class charged at entry and trued up afterwards, or a second post-outcome bucket shaping only future admission, remain unspecified candidates rather than designs. A caller can still consume admission budget with expensive refusals; that residual is bounded by attacker cost rather than by the limiter, and it is recorded in Out of scope with that reasoning. What is forbidden here is closing the gap by widening the bucket key toward outcome classes to make the two documents agree — the documents are reconciled by the amendment the decision issue owns, not by this code.
Departure 2 — the caller answer. Ratified; recorded here as the reasoning the RFC amendment carries. The RFC's rate-limit decision says a throttled invocation returns RegistrarBusy { retry_after } — a retryable error — and its acceptance criterion restates that in the same breath as the pre-derivation-arm test, neither of them distinguishing check points. This issue returns the throttle from admission only; a limited pre-derivation invocation gets its real permanent refusal. RegistrarBusy is the error family's only retryable member — the enroll wire issue exposes that as a classification method on the error type, true for it and false for the other five — while the refusals this arm produces are classified deterministic and permanent by RFC-D2 §4b, which terminates the attempt immediately and spends no retry budget, and by RFC-E §9, which states ServiceInstanceMismatch is not retryable because clicking again sends the same request. So answering RegistrarBusy here is a type-level contradiction, not an ergonomic preference. RFC-E §9 also requires the UI to render RegistrarBusy as an in-progress wait honouring retry_after, never as a failure, with no retry button — a malformed service_name would sit on screen as "waiting" indefinitely with its real cause unreachable. And the predecision_refusal bucket exists precisely because this arm is the cheapest, most floodable path, so answering RegistrarBusy on it would have the limiter feed the traffic it was added to damp. The amendment must name which check point produces the throttle and why the other does not, in both places the RFC states it: leaving either sentence unqualified reproduces this contradiction one layer down, in the sentence an implementer will quote.
The limited-invocation event
Both check points emit one event per limited invocation, carrying the client identity, the verb and the bucket that had no token. Publish it through a sink trait the limiter is constructed with, plus a default in-process implementation that keeps a u64 count per bucket — one for predecision_refusal, one for admission, both since daemon start — and nothing else. Two counters rather than one sum, because the sibling record issue publishes them separately on the endpoint and the two mean opposite things to an operator: a predecision_refusal count rising says someone is flooding malformed input while those callers still got their real answers, and an admission count rising says legitimate traffic is being held back and a bring-up may be stalling. A single total would have to be split again downstream, and it cannot be. This issue ships the mechanism and the counter; the sibling record issue implements the coalescing sink that turns those events into durable records and publishes the counters on the endpoint. Defining the seam here is what lets the limiter be implemented, tested and reviewed before any record shape exists.
The limiter holds exactly one sink, so say what happens when the sibling installs its own: it wraps this one rather than replacing it. The sibling's sink is constructed around the default counting sink and delegates every event to it before coalescing, so the two per-bucket counters stay the single implementation in the tree — which matters because that sibling's health member publishes exactly those counters, and a sink that displaced them would leave the member reading a counter nobody increments or force a second counting path to be written beside this one. Keep the trait's shape composable enough for that: an implementation takes another implementation and forwards to it. This issue does not build the wrapper — it builds the trait, the default counting sink, and nothing that prevents the wrapping.
Configuration and sizing
Size the limiter from the legitimate fan-out, not from "mints are rare in steady state". There is effectively one registrar identity, so "per client identity" is close to global: throttling an attacker throttles the control plane in the same bucket, and an ordinary bring-up — an onboarding wave plus several modules per host — is a burst.
Add these keys to the [registrar] configuration table, the table the audit-record work already adds audit_record_dir and the rotation bounds to. Every one is an unsigned integer; the configuration surface carries no floating-point value. A refill rate would have to be fractional to express "one token per second or slower", which drags in negative zero, NaN, infinity and precision questions that all have to be validated away; a refill interval expresses the same rates as plain integers and has none of them.
rate_limit_admission_burst — u32, default 512.
rate_limit_admission_refill_interval_ms — u32 milliseconds per token, default 500 (two tokens per second).
rate_limit_predecision_refusal_refill_interval_ms — u32 milliseconds per token, default 1000 (one token per second).
Validation rejects zero for all four keys, with the reason documented: a zero burst throttles the first legitimate mint, and a zero refill interval is an unbounded token supply that disables the limiter silently.
A negative or non-integer TOML value cannot deserialize into an unsigned integer at all. Surface that as a named configuration error identifying the key, in the same shape as the table's existing validation failures, rather than letting the deserialization error reach the operator raw or letting a load failure panic. Loading a [registrar] table that carries unrelated keys must leave them untouched — the audit-record work's audit_record_dir and rotation bounds and the reserved store's audit_store_* keys are already there, and more arrive later from outside this group on no fixed schedule. This issue lands its four keys first within its own group, so the sibling record issue's rate_limit_coalesce_window_seconds is a later addition rather than something that may already be present: what this issue owes it is a loader and a validator that neither reject nor clobber a [registrar] key they do not know, so that key can join the table without touching anything written here.
The sizing formula, to be stated in the shipped documentation and not only in code:rate_limit_admission_burst >= wave_hosts × modules_per_host, where wave_hosts is the largest number of hosts an operator brings up at once and modules_per_host the largest number of components on one host. The shipped default assumes the reference deployment 64 hosts × 8 modules = 512 mints in one wave, which is where 512 comes from. A wave larger than the burst still completes rather than being refused, taking (mints − burst) × rate_limit_admission_refill_interval_ms / 1000 extra seconds. Legitimate pre-derivation refusals are operator typos arriving one at a time, hence the much smaller refusal budget.
The documentation must show the deployment's own numbers run through that formula, not only the shipped defaults, so an operator whose fleet is larger than the reference can see that they must raise the burst before their first bring-up rather than after it.
Document the two buckets and what their isolation does and does not guarantee, the sizing arithmetic with the deployment's own numbers, and the four configuration keys with their defaults, in bothdocs/en/ and docs/ko/, on the existing pages beside the audit-record documentation, with no mkdocs.yml nav change.
Acceptance criteria
Gate, checked before any code is written: the condition the Status section states — the sibling decision issue's RFC amendment has landed carrying both the ratified bucket shape and the ratified admission-only caller answer — is met, and that section's editorial step has been applied to this issue: Status removed, Departure 1's charge-point reasoning folded into the Scope bullet defining the bucket key, and the departures section and both its subsections deleted. An amendment that transcribes one ruling while leaving the other unqualified does not satisfy this gate, and no criterion below may be implemented while it is unmet.
A single client identity flooding the pre-derivation arm is limited: after the refusal burst is spent, further invocations on that arm write neither an intent nor an outcome record, and each emits exactly one limited-invocation event.
That flood does not starve legitimate mints: a test drives the predecision_refusal bucket to empty and asserts a concurrent post-derivation mint is still admitted and still recorded.
On the pre-derivation arm the limiter runs before the intent write, and a suppressed invocation still returns the caller's real permanent refusal rather than the throttle.
On the post-derivation arm the limiter runs at entry, before the intent write and before any OpenBao call; a test drains the admission bucket and asserts the next invocation returns the typed retryable throttle, performs no OpenBao call and writes no record.
The throttle is never returned for a determined outcome: a test asserts a pre-derivation refusal returns its permanent reason whether or not the limiter suppressed its records, and that the throttle variant is distinguishable in the typed outcome surface from every permanent unavailability variant.
The retry payload is an unsigned whole-second duration of at least 1, never an absolute timestamp, and its value tracks the admission bucket's accrual against the configured refill interval.
The limiter is a token bucket driven by a monotonic clock: a bucket drained at a boundary does not immediately refill to a second full burst, and a wall-clock jump adds no tokens.
A bucket starts full and holds no persisted state: a freshly created limiter admits a full burst before throttling, and a drained bucket recreated after a restart starts full again. No entry is ever removed from the map while it is below its full burst.
Buckets are keyed on a value stable across connections: two sequential connections from one caller spend the same bucket rather than getting a fresh full budget, so the limiter cannot be reset by reconnecting.
The map is capped by a file-top constant counted in bucket entries (256, the reference 64 identities × four buckets): driving far more distinct client identities than the cap allows leaves the entry count at or below it, reclaims only entries at full burst, and never removes a drained entry.
The miss path is total: with the map at the cap and every entry drained, an invocation for an absent key is charged against the overflow bucket for its (verb, bucket) — including when the map holds no entry carrying that pair — and is throttled or suppressed on that bucket's tokens rather than admitted unlimited. The four overflow buckets are sized from the same keys as the map entries and are not counted against the cap.
The bucket enum has exactly two values and is exported for the sibling record issue; no bucket key, anywhere, carries the verb layer's fine-grained refusal reason.
The four rate_limit_* keys are unsigned integers that load with the documented defaults; validation rejects a zero value for each with an error naming the key, and a negative or non-integer value produces the same named configuration error rather than a raw deserialization failure or a panic. A [registrar] table carrying unrelated keys — those already in the tree and one standing in for the sibling's later rate_limit_coalesce_window_seconds — loads without rejecting them and leaves their values untouched, so a later key can join the table without this issue's loader being revisited.
The limiter is constructed with a limited-invocation sink; the shipped default sink counts limited invocations since daemon start as a u64per bucket, readable in process, and a test floods each bucket separately and asserts only that bucket's counter moves. The trait admits a wrapping implementation — one that takes another sink and forwards to it — so the sibling record issue can add coalescing without displacing these counters; a test drives the limiter through a trivial forwarding wrapper around the default sink and asserts both counters still move.
The documented sizing includes the wave_hosts × modules_per_host arithmetic and the deployment's own numbers, and a realistic bring-up burst (an onboarding wave plus several modules per host) passes without being throttled.
The buckets, their isolation boundary, the sizing and the four keys are documented in bothdocs/en/ and docs/ko/, on the existing pages, with no mkdocs.yml nav change.
cargo clippy is warning-free and cargo fmt --check passes.
Constraints
Implementation does not begin until the gate above is met. While the RFC and this specification disagree, the correct action is to wait for the decision issue's amendment to land carrying both halves, not to pick whichever reading makes the code compile.
The limiter must not be sized by intuition about steady state. Legitimate bring-up bursts must pass; the sizing rationale must be written down where an operator can find it.
Both check points must sit before the invocation's intent write. A limiter that runs after it cannot suppress the record it exists to suppress, and on the pre-derivation arm it would make the intent record the amplifier for the exhaustion the limiter is preventing.
Do not key a bucket on the verb layer's fine-grained refusal reason. It is unknown at both check points, and a key that varied with it would let a caller multiply its budget by varying its refusal.
The throttle must never be substituted for an outcome the daemon has already determined, and must not be expressed by overloading an existing variant with a boolean.
Do not weaken the "unrecordable intent implies refused" rule the audit-record work establishes. Suppression here is a deliberate skip on an arm that creates nothing, decided before the write is attempted; it is not a failed write, and it never makes an accepted mint unrecorded.
Do not restructure the verbs' control flow, their two arms, their locks or their refusal taxonomy. The limiter runs at points those already define, and a suppressed pre-derivation invocation still runs the arm's checks and returns its usual answer. Do not interpret or re-derive the caller identity either — it is opaque, and a bucket key that assumes a certificate identity breaks the pre-mTLS caller.
Keep the configuration surface free of floating-point values; express rates as integer millisecond intervals.
Do not hold a bucket lock across an OpenBao call or a record write. The limiter's decision is taken and released at the check point; everything after it runs under the locks the verbs already define.
No unwrap() in production code; no [] indexing.
Out of scope
Amending the RFC. Both departures above are recorded here as rationale only; the RFC edit belongs entirely to the sibling decision issue, including the amendment discharging the now-ratified bucket shape — §5.6 states the per-(verb, outcome-class) shape as a guarantee, so until that edit lands the document asserts something this code will not deliver. This issue changes no RFC text under any outcome.
The counted record and everything about it: the coalescing window and its configuration key, the record variant, its encoding and fixture, the reader classification amendments, and the field bound on the identity it carries. This issue emits events through a sink; the sibling record issue turns them into records.
The throttle's caller-facing wire identifier, its payload spelling and its golden fixture, and the registrar_health member that publishes the limited-invocation count. This issue produces a typed, retryable outcome inside the daemon; the sibling record issue maps and publishes it.
A per-(component, host)ceiling on how many instances a deployment may legitimately run — a different object from a rate limit, and deliberately deferred.
Isolating legitimate mints from a flood of post-derivation refusals. Those consume admission budget by construction, because nothing distinguishes them from a real mint before the intent write — the ratified bucket shape is the maximal refinement knowable where the limiter is charged. This is the ruling's stated residual and it carries its own bound: an attacker driving it pays OpenBao work per attempt, so the path is limited by attacker cost rather than by this mechanism. Recorded here with that reasoning rather than as a bare accepted weakening.
Defining the pre-derivation arm, the locks, the request id or the caller-identity parameter — all owned by the verbs issue and consumed here.
Defining the record encoding, the rotation bounds, the anomaly scan or audit_record_dir itself.
The reserved audit store in every part: its configuration keys, layout, Compose override, capacity probe, headroom arithmetic and low-water alarm.
Test plan
Pre-derivation flood test: after the burst, neither record phase is written, one event is emitted per invocation, and the caller still receives its real permanent refusal.
Bucket-isolation test: drain the predecision_refusal bucket and assert a post-derivation mint is still admitted and recorded.
Ordering tests: on the pre-derivation arm the limiter precedes the intent write; on the post-derivation arm it precedes both the intent write and any OpenBao call, asserted through a seam that fails if OpenBao is reached.
Admission-throttle test asserting the typed retryable outcome with a retry value of at least 1, no OpenBao call, no record, and one event carrying the admission bucket.
Token-bucket behaviour test: monotonic-clock refill, no double budget across a boundary, no tokens from a wall-clock jump, a fresh bucket starting full, and no removal of an entry below its burst.
Bucket-key stability test: two sequential connections from one caller spend the same bucket.
Bounded-map tests: driving far more distinct identities than the cap keeps the entry count at or below it; a map at the cap holding a full entry reclaims it and admits the new key; a map at the cap with every entry drained charges the overflow bucket for the invocation's (verb, bucket), covering the case where no map entry carries that pair; and no path removes an entry below its full burst.
Config tests: the four rate_limit_* keys load with their defaults; a zero value for each, and a negative and a non-integer value for one, each fail validation with an error naming the key; unrelated [registrar] keys survive the load.
Sink tests, three cases: the default sink's two per-bucket counts each track a flood driven at their own bucket and neither moves for the other's; the limiter functions with a sink that does nothing; and the limiter driven through a trivial forwarding wrapper around the default sink still advances both counters, which is the composability the sibling record issue's coalescing sink depends on.
Legitimate-burst test asserting a realistic bring-up wave is not throttled.
Tests use tempfile::tempdir() and never a fixed path.
Dependencies
Blocked on the sibling decision issue — Settle the limiter's bucket shape against the RFC — which carries the RFC amendment for both rulings this specification rests on, the bucket key and the caller answer. Both are already decided; what is still owed is the document edit, and until it lands the accepted RFC states two guarantees this code will not deliver. The unblocking condition is the one the Status section states and the acceptance gate checks: that issue's RFC amendment lands carrying both. Neither transcription suffices on its own, and this issue states that condition in exactly one place to keep the two from drifting apart. The Status section also carries the editorial step to apply at that moment; it is purely editorial, and no Scope substance changes. Also depends on the two-phase audit-record integration for the intent/outcome ordering these check points must precede and for the verb arms they sit at the entry of, and through it on the verbs issue for the two arms, their locks and the opaque caller identity the buckets are keyed on. Part of the registrar verb rate-limiting umbrella.
Pointers
src/daemon.rs::run_daemon:68 — the daemon loop the endpoint and its ticks live in
src/openbao.rs:570 (verify_audit_file) — the mandatory audit device whose exhaustion this limiter exists to bound
openbao/openbao.hcl — the audit stanza that makes that device mandatory
src/config.rs — the configuration load and validation shape the four rate_limit_* keys follow
docs/en/operations.md:95 ("Audit logging") and docs/ko/operations.md:87 ("감사 로깅") — the audit-logging sections the limiter documentation sits beside
docs/rfcs/0001-registrar-role-and-non-self-propagation.md §5.6 (the two rate-limit decisions and the pre-derivation-arm decision), §6 (the verb rate-limit acceptance criterion) and §7 (the decomposition entry that restates both) — the four places the disputed language lives
Bound both registrar verbs with two token buckets
Status — both questions settled; blocked only on the RFC amendment
Do not implement it as filed yet, but nothing in it is unsettled. The specification below departed from the accepted RFC in two independent places, and the RFC's owner has now ratified both exactly as specified here: the bucket key as per-
(client identity, verb, bucket)buckets over the two-value enum, and the caller answer as the throttle from the admission check point only, with a limited pre-derivation invocation keeping its real permanent refusal. Everything from the Context down is authorised. What remains is bookkeeping with teeth — the sibling decision issue must land the RFC amendment carrying both rulings, because until it does the accepted document still states guarantees this code will not deliver.(verb, outcome-class)buckets "so refusals throttle without starving accepted mints"; this issue's two-value shape stands, and Departure 1 below now records the ratified reasoning rather than a proposal. Nothing in the Scope changes.RegistrarBusy { retry_after }, and its pre-derivation-arm decision calls a records-suppressed invocation on that arm "throttled" — so the RFC as written implies that answer on both check points. This issue returns the throttle from the admission check point only, and that split is the ruling. Departure 2 below records the ratified reasoning. Nothing in the Scope changes.The unblocking procedure. This section is the whole of it — no other artifact needs to be consulted to know what happens when the amendment lands.
(verb, outcome-class)bucket shape as a guarantee, and §5.6 and §6 both state theRegistrarBusyanswer without qualifying the check point — so until the amendment lands the accepted document asserts two things this code will not deliver. Neither half alone unblocks this issue, and rulings recorded only in the decision issue's discussion, without the RFC edit, do not either.rate_limit_*keys, the isolation guarantee below, the admission-only throttle, and a limited pre-derivation invocation keeping its real permanent refusal while its records are suppressed. Do not widen the bucket key toward outcome classes, do not widen the throttle to the pre-derivation arm, and do not treat a later reading of the unamended RFC sentences as reopening either — those sentences are the amendment's to fix.Context
A record is now required for every registrar verb invocation, including refusals. That closes a detection gap and opens a resource one: there is no limit anywhere on either verb, so a compromised caller — the exact actor this whole mechanism exists for — can flood refused invocations and grow the record store without bound. The cheapest refusals are the ones rejected before any OpenBao work, so the flood path is also the free path.
The consequence is not merely a large file. When the filesystem fills, every subsequent mint is refused, so fleet-wide enrollment stops. Worse, the OpenBao file audit device this design builds on is mandatory —
openbao/openbao.hcldeclares anaudit { type = "file" ... }stanza andbootroot initrefuses to proceed without it (src/openbao.rs::verify_audit_file) — and OpenBao fails requests when it cannot write its audit device. So a full disk can stop OpenBao serving, which is the renewal path for every certificate in the deployment, on the one host that must not be restarted. "Mints are rare in steady state" is true of legitimate mints and irrelevant to the attacker.This issue builds the mechanism that bounds the rate: the buckets, where they are charged, and what each check point tells its caller. Two words are used precisely throughout and are not synonyms: an invocation is limited when this issue suppressed its audit records, which happens at both check points, and throttled when it is limited at the admission check point and therefore receives the retryable throttle outcome. Every throttled invocation is limited; the reverse does not hold. A sibling issue turns the invocations this one suppresses into counted records, amends the record reader for them, and publishes the counter on the endpoint. A third, unrelated issue bounds the space those records land in. A per-
(component, host)ceiling on how many instances a deployment may legitimately run is a fourth object again, and is deliberately deferred.Scope
Two buckets, both consulted before any durable write
The limiter has exactly two check points, and each sits before the invocation's intent record is written. This ordering is forced, not stylistic: the audit-record work writes the intent at arm entry — on the pre-derivation arm, before that arm's checks run — so at every point where the limiter can still prevent a durable write, the specific refusal reason is not yet known. The buckets are therefore keyed on where the invocation is, not on what it will turn out to be.
(client identity, verb, bucket), wherebucketis an enum with exactly two values, defined here and exported for the sibling record issue to reuse:predecision_refusal— the verb layer's pre-derivation arm: the arm entered for aservice_nameorhostthat is not a DNS label, or a component with no multiplicity entry. Every invocation on this arm is a refusal by construction, it performs no OpenBao work, and it is the cheapest thing a caller can make the daemon do.admission— every invocation that reaches the post-derivation arm and may therefore reach OpenBao, whatever it turns out to be.(client identity, verb, bucket)key, and the cap is a file-top constant on the number of entries, not on the number of identities:256, which is the reference deployment's64identities × the four buckets each identity can create. Entries are what consume the memory, and counting them keeps the cap in the same unit as every removal rule below — a cap counted in identities cannot be honoured by removing a bucket at all, since an identity holding one drained bucket and three full ones is not removable and freeing its three full ones frees no identity slot. There is effectively one registrar identity, so the cap is never reached in normal operation; it exists because the identity-stability failure above must not become memory exhaustion on the bootroot host.(verb, bucket).(verb, bucket)pair, created with the limiter, sized from the same configured burst and refill interval as the map entries for thatbucket, never reclaimed, and not counted against the cap. They exist to make the miss path total: whatever the map holds, there is always exactly one bucket to charge, so no invocation is ever admitted unlimited for want of a slot. Charging "some existing entry for the same(verb, bucket)" would not be total — a saturated map need not contain that pair at all, which is reachable whenever traffic is mixed and buckets are created lazily — and it would spend one identity's budget on another's traffic by an unspecified choice of victim.Check point 1 — pre-derivation arm entry, before the intent write. Consult and charge the
predecision_refusalbucket at the moment the arm is entered and before its intent record is written. With a token available, the arm proceeds exactly as the audit-record work specifies: intent, checks, outcome, paired records. With the bucket empty, skip both record writes, emit one limited-invocation event (below), and otherwise let the arm run unchanged — its checks are pure input validation with no I/O — so the caller receives its real, permanent refusal, not a throttle. This is the free-flood path, and it is the only place where records are suppressed and the caller is still given its real answer.Check point 2 — post-derivation arm entry, before the intent write and before any OpenBao work. Consult and charge the
admissionbucket. With the bucket empty the invocation is not attempted at all: it returns the typed throttle outcome below, writes no intent and no outcome, and emits one limited-invocation event.The throttle outcome
RegistrarBusyis the family's only retryable member, and this arm's refusals are classified permanent by the merged documents the sibling record issue maps onto. Reporting one as retryable would also make a broken caller retry a request that can never succeed, and that retry storm is itself a flood. So throttling never converts a determined outcome into a retryable one; on the pre-derivation arm the limiter suppresses the record, never the answer.admissionbucket holds one token, derived from the bucket's elapsed accrual against the admission refill interval and rounded up to whole seconds, with a floor of one second, so a value of0can never invite an immediate retry. It is deterministic; document that a caller with several outstanding requests should jitter its own retries.What the two buckets do and do not guarantee — stated plainly, because the boundary matters. A flood on the
predecision_refusalpath cannot consumeadmissionbudget, so the path an attacker can drive for free, with no valid input and no OpenBao cost, cannot starve legitimate mints. That is the isolation this design buys, and it is the flood the Context describes. A caller that can produce well-formed, derivable requests can still consumeadmissionbudget with invocations that end in refusal, because at admission nothing distinguishes them from a real mint. That is bounded by attacker cost rather than by the limiter — every such attempt spends OpenBao work as well as an admission token — but it is not isolated from legitimate mints, and the documentation must say so rather than imply a guarantee the design does not provide. State the delivered guarantee positively while doing so: the isolation above is exactly the path §5.6's own threat rationale names when it callsServiceInstanceMismatchthe cheapest refusal, and both of that error's causes are pre-derivation. Bounding the expensive path more tightly would require classifying the outcome before the intent write, which the record ordering forbids.Why this specification departs from the accepted RFC
Two departures, not one, and the second is not a consequence of the first — either bucket shape composes with either caller answer. Both have been ruled on, and each is recorded below as the settled reasoning the RFC amendment carries. Neither is an open question, and neither may be reopened, renegotiated or closed differently inside this issue. This entire section is temporary. The Status section's editorial step deletes it — both subsections — once the amendment lands, after folding Departure 1's charge-point reasoning into the Scope bullet that defines the bucket key, so that reasoning is kept while the departure framing goes.
Departure 1 — the bucket key. Ratified; recorded here as the reasoning the RFC amendment carries. The RFC specifies per-
(verb, outcome-class)buckets "so refusals throttle without starving accepted mints" — a stronger guarantee than the two buckets above deliver, since it would isolate every class of refusal from legitimate mints rather than only the pre-derivation class. The accurate reason is not that the RFC's shape is unimplementable, and this issue does not claim that. It is that the two-value enum is the maximal refinement knowable at the point the limiter is charged: a pre-derivation refusal is knowable there, because that arm fires only on refusals and label validation and multiplicity-class resolution both precede the check point, while the outcome of an admitted invocation is not, because spec-conflict, name-collision and host-mismatch are settled by OpenBao work that follows the intent write. Keying on the outcome class would therefore require charging after the durable write the limiter exists to prevent, or predicting the outcome, and no third shape is specified today — a provisional class charged at entry and trued up afterwards, or a second post-outcome bucket shaping only future admission, remain unspecified candidates rather than designs. A caller can still consume admission budget with expensive refusals; that residual is bounded by attacker cost rather than by the limiter, and it is recorded in Out of scope with that reasoning. What is forbidden here is closing the gap by widening the bucket key toward outcome classes to make the two documents agree — the documents are reconciled by the amendment the decision issue owns, not by this code.Departure 2 — the caller answer. Ratified; recorded here as the reasoning the RFC amendment carries. The RFC's rate-limit decision says a throttled invocation returns
RegistrarBusy { retry_after }— a retryable error — and its acceptance criterion restates that in the same breath as the pre-derivation-arm test, neither of them distinguishing check points. This issue returns the throttle from admission only; a limited pre-derivation invocation gets its real permanent refusal.RegistrarBusyis the error family's only retryable member — the enroll wire issue exposes that as a classification method on the error type, true for it and false for the other five — while the refusals this arm produces are classified deterministic and permanent by RFC-D2 §4b, which terminates the attempt immediately and spends no retry budget, and by RFC-E §9, which statesServiceInstanceMismatchis not retryable because clicking again sends the same request. So answeringRegistrarBusyhere is a type-level contradiction, not an ergonomic preference. RFC-E §9 also requires the UI to renderRegistrarBusyas an in-progress wait honouringretry_after, never as a failure, with no retry button — a malformedservice_namewould sit on screen as "waiting" indefinitely with its real cause unreachable. And thepredecision_refusalbucket exists precisely because this arm is the cheapest, most floodable path, so answeringRegistrarBusyon it would have the limiter feed the traffic it was added to damp. The amendment must name which check point produces the throttle and why the other does not, in both places the RFC states it: leaving either sentence unqualified reproduces this contradiction one layer down, in the sentence an implementer will quote.The limited-invocation event
Both check points emit one event per limited invocation, carrying the client identity, the verb and the bucket that had no token. Publish it through a sink trait the limiter is constructed with, plus a default in-process implementation that keeps a
u64count per bucket — one forpredecision_refusal, one foradmission, both since daemon start — and nothing else. Two counters rather than one sum, because the sibling record issue publishes them separately on the endpoint and the two mean opposite things to an operator: apredecision_refusalcount rising says someone is flooding malformed input while those callers still got their real answers, and anadmissioncount rising says legitimate traffic is being held back and a bring-up may be stalling. A single total would have to be split again downstream, and it cannot be. This issue ships the mechanism and the counter; the sibling record issue implements the coalescing sink that turns those events into durable records and publishes the counters on the endpoint. Defining the seam here is what lets the limiter be implemented, tested and reviewed before any record shape exists.Configuration and sizing
[registrar]configuration table, the table the audit-record work already addsaudit_record_dirand the rotation bounds to. Every one is an unsigned integer; the configuration surface carries no floating-point value. A refill rate would have to be fractional to express "one token per second or slower", which drags in negative zero, NaN, infinity and precision questions that all have to be validated away; a refill interval expresses the same rates as plain integers and has none of them.rate_limit_admission_burst—u32, default512.rate_limit_admission_refill_interval_ms—u32milliseconds per token, default500(two tokens per second).rate_limit_predecision_refusal_burst—u32, default32.rate_limit_predecision_refusal_refill_interval_ms—u32milliseconds per token, default1000(one token per second).[registrar]table that carries unrelated keys must leave them untouched — the audit-record work'saudit_record_dirand rotation bounds and the reserved store'saudit_store_*keys are already there, and more arrive later from outside this group on no fixed schedule. This issue lands its four keys first within its own group, so the sibling record issue'srate_limit_coalesce_window_secondsis a later addition rather than something that may already be present: what this issue owes it is a loader and a validator that neither reject nor clobber a[registrar]key they do not know, so that key can join the table without touching anything written here.rate_limit_admission_burst >= wave_hosts × modules_per_host, wherewave_hostsis the largest number of hosts an operator brings up at once andmodules_per_hostthe largest number of components on one host. The shipped default assumes the reference deployment 64 hosts × 8 modules = 512 mints in one wave, which is where512comes from. A wave larger than the burst still completes rather than being refused, taking(mints − burst) × rate_limit_admission_refill_interval_ms / 1000extra seconds. Legitimate pre-derivation refusals are operator typos arriving one at a time, hence the much smaller refusal budget.docs/en/anddocs/ko/, on the existing pages beside the audit-record documentation, with nomkdocs.ymlnav change.Acceptance criteria
predecision_refusalbucket to empty and asserts a concurrent post-derivation mint is still admitted and still recorded.admissionbucket and asserts the next invocation returns the typed retryable throttle, performs no OpenBao call and writes no record.256, the reference64identities × four buckets): driving far more distinct client identities than the cap allows leaves the entry count at or below it, reclaims only entries at full burst, and never removes a drained entry.(verb, bucket)— including when the map holds no entry carrying that pair — and is throttled or suppressed on that bucket's tokens rather than admitted unlimited. The four overflow buckets are sized from the same keys as the map entries and are not counted against the cap.rate_limit_*keys are unsigned integers that load with the documented defaults; validation rejects a zero value for each with an error naming the key, and a negative or non-integer value produces the same named configuration error rather than a raw deserialization failure or a panic. A[registrar]table carrying unrelated keys — those already in the tree and one standing in for the sibling's laterrate_limit_coalesce_window_seconds— loads without rejecting them and leaves their values untouched, so a later key can join the table without this issue's loader being revisited.u64per bucket, readable in process, and a test floods each bucket separately and asserts only that bucket's counter moves. The trait admits a wrapping implementation — one that takes another sink and forwards to it — so the sibling record issue can add coalescing without displacing these counters; a test drives the limiter through a trivial forwarding wrapper around the default sink and asserts both counters still move.wave_hosts × modules_per_hostarithmetic and the deployment's own numbers, and a realistic bring-up burst (an onboarding wave plus several modules per host) passes without being throttled.docs/en/anddocs/ko/, on the existing pages, with nomkdocs.ymlnav change.cargo clippyis warning-free andcargo fmt --checkpasses.Constraints
unwrap()in production code; no[]indexing.Out of scope
(verb, outcome-class)shape as a guarantee, so until that edit lands the document asserts something this code will not deliver. This issue changes no RFC text under any outcome.registrar_healthmember that publishes the limited-invocation count. This issue produces a typed, retryable outcome inside the daemon; the sibling record issue maps and publishes it.(component, host)ceiling on how many instances a deployment may legitimately run — a different object from a rate limit, and deliberately deferred.audit_record_diritself.Test plan
predecision_refusalbucket and assert a post-derivation mint is still admitted and recorded.admissionbucket.(verb, bucket), covering the case where no map entry carries that pair; and no path removes an entry below its full burst.rate_limit_*keys load with their defaults; a zero value for each, and a negative and a non-integer value for one, each fail validation with an error naming the key; unrelated[registrar]keys survive the load.tempfile::tempdir()and never a fixed path.Dependencies
Blocked on the sibling decision issue — Settle the limiter's bucket shape against the RFC — which carries the RFC amendment for both rulings this specification rests on, the bucket key and the caller answer. Both are already decided; what is still owed is the document edit, and until it lands the accepted RFC states two guarantees this code will not deliver. The unblocking condition is the one the Status section states and the acceptance gate checks: that issue's RFC amendment lands carrying both. Neither transcription suffices on its own, and this issue states that condition in exactly one place to keep the two from drifting apart. The Status section also carries the editorial step to apply at that moment; it is purely editorial, and no Scope substance changes. Also depends on the two-phase audit-record integration for the intent/outcome ordering these check points must precede and for the verb arms they sit at the entry of, and through it on the verbs issue for the two arms, their locks and the opaque caller identity the buckets are keyed on. Part of the registrar verb rate-limiting umbrella.
Pointers
src/daemon.rs::run_daemon:68— the daemon loop the endpoint and its ticks live insrc/openbao.rs:570(verify_audit_file) — the mandatory audit device whose exhaustion this limiter exists to boundopenbao/openbao.hcl— the audit stanza that makes that device mandatorysrc/config.rs— the configuration load and validation shape the fourrate_limit_*keys followdocs/en/operations.md:95("Audit logging") anddocs/ko/operations.md:87("감사 로깅") — the audit-logging sections the limiter documentation sits besidedocs/rfcs/0001-registrar-role-and-non-self-propagation.md§5.6 (the two rate-limit decisions and the pre-derivation-arm decision), §6 (the verb rate-limit acceptance criterion) and §7 (the decomposition entry that restates both) — the four places the disputed language lives