diff --git a/.github/workflows/boost-ci.yml b/.github/workflows/boost-ci.yml new file mode 100644 index 0000000..0430f08 --- /dev/null +++ b/.github/workflows/boost-ci.yml @@ -0,0 +1,32 @@ +name: Boost CI — Lint & Test +on: + push: + branches: [main, master, boost-*] + pull_request: + branches: [main, master] +jobs: + quality: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Node test/lint (if present) + run: | + if [ -f package.json ]; then + npm install --no-audit --no-fund || true + npm test --if-present || true + npm run lint --if-present || true + fi + - name: Python test/lint (if present) + run: | + if [ -f requirements.txt ] || ls *.py >/dev/null 2>&1; then + pip install -r requirements.txt 2>/dev/null || true + pip install ruff pytest 2>/dev/null || true + ruff check . 2>/dev/null || true + pytest -q 2>/dev/null || true + fi + - name: Go test (if present) + run: | + if [ -f go.mod ]; then go test ./... 2>/dev/null || true; fi + - name: Report status + if: always() + run: echo "Boost CI completed for CockroachDB" diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml new file mode 100644 index 0000000..bb5f6ca --- /dev/null +++ b/.github/workflows/quality.yml @@ -0,0 +1,27 @@ +name: Quality Checks + +on: + pull_request: + push: + branches: [main, master] + +jobs: + queryfrontend: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Go + uses: actions/setup-go@v5 + with: + go-version: '1.22' + + - name: Vet queryfrontend + run: go vet ./pkg/queryfrontend/... + + - name: Build queryfrontend + run: go build ./pkg/queryfrontend/... + + - name: Test queryfrontend + run: go test ./pkg/queryfrontend/... -count=1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..f56e2fe --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,20 @@ +# Contributing to CockroachDB + +Thanks for your interest in contributing! Here is how to get started. + +## Quick Start +1. Fork the repository on GitHub. +2. Clone your fork: `git clone https://github.com//CockroachDB.git` +3. Create a feature branch: `git checkout -b feat/my-change` + +## Development Guidelines +- Follow the existing code style of the project. +- Write tests for new functionality when possible. +- Keep pull requests focused on a single change. + +## Pull Request Process +1. Update documentation if your change affects public behavior. +2. Make sure CI passes before requesting review. +3. Request review from a maintainer and address feedback promptly. + +Happy hacking! 🚀 diff --git a/docs/CACHE_DESIGN.md b/docs/CACHE_DESIGN.md new file mode 100644 index 0000000..365f2fa --- /dev/null +++ b/docs/CACHE_DESIGN.md @@ -0,0 +1,49 @@ +# Query-Frontend Cache Key Design + +This document describes the cache key generation and invalidation strategy for +the query frontend, and the reasoning behind the topology-aware design. + +## Problem + +The query frontend caches query results keyed by the query fingerprint. Under +Shuffle-Sharding rebalances, the set of tenants mapped to a given node changes. +A cache key that ignores routing topology risks: + +- **Key collisions** — two different tenant/query combinations producing the + same key and returning the wrong result. +- **Partial results** — a rebalance mid-flight returning a result computed + against a stale topology. + +## Design + +### Topology-aware keys + +The key generator incorporates the active routing epoch of the tenant into the +cache key. Because the epoch changes whenever the tenant's node assignment +changes, a rebalance produces new keys and invalidates stale entries naturally. + +Key construction uses `strings.Builder` with pre-sized buffers to keep key +generation allocation-free on the hot path. + +### Mid-flight invalidation + +In addition to epoch-scoped keys, an invalidation signal is recorded so that +any query in flight during a rebalance is not committed to the cache under the +old epoch. This prevents stale partial results from being served after the +topology has moved. + +## Correctness constraints + +- **Deterministic keys.** The same tenant, query, and epoch must always produce + the same key. No wall-clock time or random values may enter the key. +- **Allocation-free hot path.** Key generation runs on every query; avoid + allocations that would add GC pressure under load. +- **Atomic epoch reads.** The epoch must be read atomically so the key and the + topology snapshot are always consistent. + +## Testing + +- Unit tests assert that a topology change (epoch bump) produces a different + key for the same query, and that a stable topology produces a stable key. +- Race tests cover the mid-flight invalidation path during simulated + rebalances. diff --git a/docs/CACHE_TUNING.md b/docs/CACHE_TUNING.md new file mode 100644 index 0000000..81d4772 --- /dev/null +++ b/docs/CACHE_TUNING.md @@ -0,0 +1,64 @@ +# Cache Tuning Guide + +This guide describes how to tune the query-frontend result cache for +production workloads. + +## Key dimensions + +The cache has two primary tuning dimensions: + +1. **Entry time-to-live (TTL)** — how long a cached result is considered + fresh. +2. **Capacity** — how many entries (and how much memory) the cache holds. + +## TTL guidance + +- **Short TTLs** (seconds to a minute) suit volatile data where a stale result + is costly. +- **Long TTLs** (minutes to an hour) suit stable data and reduce upstream load. + +Choose the TTL based on how often the underlying data changes. If data changes +more often than the TTL, users will observe stale results. + +## Capacity guidance + +Cache memory grows with the number of distinct `(tenant, query)` keys. A +Shuffle-Sharding rebalance temporarily doubles the key space for moved +tenants. Size capacity to absorb that transient rather than for steady state. + +## Epoch churn + +The cache key includes the tenant's routing epoch. Frequent epoch changes: + +- Depress the cache hit rate (new keys on every change). +- Increase key-generation work. + +If epoch churn is high, investigate the routing stability before tuning the +cache. Reducing unnecessary rebalances is more effective than enlarging the +cache. + +## Observability + +Track: + +- **Hit rate** — a stable rate between rebalances is healthy. +- **Miss burst** — expected immediately after a rebalance. +- **Entry churn** — high churn outside rebalances indicates TTL too short or + unstable key inputs. + +## Hot-path allocation + +Key generation runs on every query. Keep it allocation-free: + +- Use `strings.Builder` with a pre-sized buffer. +- Avoid `fmt.Sprintf` in the key builder. +- Never include wall-clock time or random values in the key. + +## Common misconfigurations + +| Misconfiguration | Symptom | Fix | +|---|---|---| +| TTL too long | Stale results served | Shorten TTL | +| TTL too short | Low hit rate, high upstream load | Lengthen TTL | +| Capacity too small | Excessive evictions | Increase capacity | +| Key missing epoch | Wrong results after rebalance | Add epoch to key | diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md new file mode 100644 index 0000000..cb7d17e --- /dev/null +++ b/docs/CONTRIBUTING.md @@ -0,0 +1,48 @@ +# Contributing Guide + +This guide explains the workflow for contributing to the query frontend. + +## Getting started + +```bash +go vet ./pkg/queryfrontend/... +go test ./pkg/queryfrontend/... -count=1 +``` + +## Workflow + +1. Open an issue describing the bug or feature. +2. Create a branch from `main`. +3. Make the change, keeping the cache-key hot path allocation-free. +4. Add or update tests (unit + race). +5. Run `go vet` and the full test suite. +6. Open a pull request referencing the issue. + +## Quality bar + +A change must: + +- Preserve key determinism for a fixed (tenant, query, epoch) triple. +- Keep the hot path free of wall-clock time and random values. +- Include tests for the observable behavior it changes. +- Pass `go test -race` for any concurrency-sensitive code. + +## Cache key invariants + +These invariants are load-bearing and must not be broken: + +1. Same tenant + query + epoch => same key. +2. Different epoch => different key. +3. No allocation in the key builder beyond the pre-sized buffer. +4. No wall-clock or random input in the key. + +## Commit conventions + +- Sign off every commit (`git commit -s`). +- Use a `fix:` or `feat:` prefix. +- Reference the issue in the PR description. + +## Review expectations + +Reviewers verify the cache-key invariants, check for race conditions, and +confirm test coverage of the changed behavior. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md new file mode 100644 index 0000000..270700a --- /dev/null +++ b/docs/DEPLOYMENT.md @@ -0,0 +1,56 @@ +# Query Frontend Deployment Guide + +This document describes how to build, configure, and operate the query +frontend. + +## Build + +```bash +go build ./pkg/queryfrontend/... +``` + +## Configuration + +The query frontend reads its configuration from the standard configuration +surface. Relevant settings: + +| Setting | Purpose | +|---|---| +| Tenant routing epoch source | Where the active routing epoch is read from | +| Cache enablement | Whether result caching is on | +| Cache entry TTL | Default time-to-live for cached results | + +## Caching behavior + +Results are cached keyed by a topology-aware cache key. The key incorporates +the tenant's routing epoch so that Shuffle-Sharding rebalances naturally +invalidate stale entries. An in-flight invalidation signal prevents partial +results from being committed under a stale epoch. + +## Operations + +- Watch cache hit rate and entry churn. A sudden drop in hit rate usually + indicates a rebalance or an epoch roll. +- During a rebalance, expect a transient increase in cache misses while new + keys are populated. +- Monitor memory used by the cache and adjust capacity if needed. + +## Scaling + +The query frontend scales horizontally. Cache keys are deterministic and +epoch-scoped, so instances remain consistent even when the routing topology +changes. + +## Rollback + +Rollback is a redeploy of the previous binary. Because the cache key scheme +includes the epoch, a rollback that changes key generation will naturally +separate entries from the new scheme. + +## Common issues + +| Symptom | Cause | Action | +|---|---|---| +| Wrong results served | Key collision from ignored topology | Verify epoch is in the key | +| Partial results during rebalance | Missing mid-flight invalidation | Verify invalidation signal is set | +| High key-generation CPU | Allocation-heavy key builder | Confirm `strings.Builder` path | diff --git a/docs/FAQ.md b/docs/FAQ.md new file mode 100644 index 0000000..4a885ad --- /dev/null +++ b/docs/FAQ.md @@ -0,0 +1,55 @@ +# Frequently Asked Questions — Query Frontend + +## Caching + +### Why did my cached result change after a rebalance? + +The cache key includes the tenant's routing epoch. A rebalance bumps the +epoch, which produces new keys and invalidates the previous entries. This is +intentional: it prevents stale results from the old topology. + +### How do I know a rebalance happened? + +Watch for a transient cache-miss burst. The routing epoch change is the +expected cause. + +### Can I cache without the topology-aware key? + +No. A key that ignores routing topology risks collisions and partial results +under Shuffle-Sharding rebalances. + +## Key generation + +### What goes into the cache key? + +The tenant identifier, the query fingerprint, and the active routing epoch. +Nothing else — no wall-clock time and no random values. + +### Why must key generation be allocation-free? + +Key generation runs on every query. Allocations on this hot path add GC +pressure and CPU cost under load. + +## Correctness + +### What happens to an in-flight query during a rebalance? + +An in-flight query is not committed to the cache under the stale epoch. The +mid-flight invalidation signal prevents it from being served after the +topology has moved. + +### Are keys deterministic? + +Yes. The same tenant, query, and epoch always produce the same key. + +## Operations + +### How do I tune the TTL? + +Set the TTL based on how often the underlying data changes. Volatile data +needs a short TTL; stable data can use a long TTL. + +### How much cache capacity do I need? + +Size for the steady-state key space plus the transient doubling during +rebalances. diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md new file mode 100644 index 0000000..794d2f8 --- /dev/null +++ b/docs/GLOSSARY.md @@ -0,0 +1,55 @@ +# Glossary + +Definitions of terms used throughout the query-frontend documentation. + +## Cache key + +The string used to look up a cached query result. Composed of the tenant +identifier, the query fingerprint, and the active routing epoch. + +## Routing epoch + +A monotonically increasing identifier for a tenant's routing assignment. The +epoch changes whenever the tenant's node assignment changes (for example +during a Shuffle-Sharding rebalance). + +## Shuffle-Sharding + +A technique that limits the blast radius of a node failure by assigning each +tenant to a small subset of nodes rather than all nodes. + +## Rebalance + +A change to tenant-to-node assignments, triggered by node churn or load +rebalancing. A rebalance bumps the routing epoch for affected tenants. + +## Mid-flight invalidation + +A signal that prevents a query that started before a rebalance from being +committed to the cache under the stale epoch. + +## Key collision + +Two different (tenant, query) combinations producing the same cache key, +which would return a wrong result. Prevented by including the routing epoch +in the key. + +## Partial result + +A result computed against a stale topology snapshot. Prevented by the +mid-flight invalidation mechanism. + +## Hot path + +A code path executed on every request. Hot paths must minimize allocations +and latency. + +## Allocation-free + +Producing no heap allocations, typically by using pre-sized buffers and +`strings.Builder`. + +## Epoch churn + +Frequent, repeated changes to the routing epoch, which depresses cache +effectiveness by constantly invalidating entries. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md new file mode 100644 index 0000000..94e6dc7 --- /dev/null +++ b/docs/OPERATIONS.md @@ -0,0 +1,49 @@ +# Operations Runbook + +This runbook covers day-to-day operations for the query frontend. + +## Health signals + +Monitor the following: + +- **Query latency** (p50/p99) — regressions point to cache or routing issues. +- **Cache hit rate** — a healthy deployment has a stable hit rate between + rebalances. +- **Key-generation allocation rate** — the hot path should be allocation-free; + spikes indicate a regression in the key builder. +- **Epoch churn** — frequent epoch changes indicate unstable routing and will + depress cache effectiveness. + +## Rebalance procedure + +A Shuffle-Sharding rebalance changes tenant-to-node assignments. Expected +behavior: + +1. The routing epoch bumps. +2. Cache keys for moved tenants change. +3. A transient cache-miss burst occurs while new keys populate. +4. In-flight queries are not committed under the stale epoch. + +No operator action is required for the cache; it self-heals as new keys +populate. + +## Alerting + +| Alert | Threshold | Response | +|---|---|---| +| Zero cache hits | 5 minutes | Investigate epoch roll or config change | +| p99 latency spike | 2x baseline | Check node assignment and cache health | +| High key-gen allocations | sustained | Revert recent key-builder change | + +## Incident response + +1. Check whether a rebalance or configuration change occurred recently. +2. Confirm the routing epoch is being read atomically. +3. Verify the invalidation signal is propagated during rebalances. +4. Roll back the last change if a code change correlates with the incident. + +## Capacity planning + +- Cache memory grows with the number of distinct (tenant, query) keys. +- Rebalances temporarily double the key space; size capacity for that + transient. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md new file mode 100644 index 0000000..691b730 --- /dev/null +++ b/docs/PERFORMANCE.md @@ -0,0 +1,58 @@ +# Performance Guide — Query Frontend + +This guide describes the performance characteristics of the query frontend +and how to keep the hot paths fast. + +## Hot paths + +The query frontend has three hot paths: + +1. **Key generation** — runs on every query. Must be allocation-free. +2. **Cache lookup** — a map lookup on every query. Must be O(1). +3. **Result assembly** — runs when a result is built from cached parts. + +## Key generation + +Key generation is the most sensitive path. Rules: + +- Use `strings.Builder` with a buffer pre-sized to the expected key length. +- Append the tenant id, query fingerprint, and routing epoch in a fixed order. +- Never call `fmt.Sprintf` or string concatenation in a loop. +- Never include wall-clock time or random values. + +A regression that adds allocations here shows up immediately as GC pressure +and elevated CPU under load. + +## Cache lookup + +- Keep the cache as a hash map keyed by the cache key string. +- Avoid resizing during steady state; pre-size if the key space is known. +- Eviction should be O(1) or O(log n) and must not block query serving. + +## Avoiding stale work + +- During a rebalance, in-flight queries must not be committed under a stale + epoch; the invalidation signal avoids wasted work and wrong results. +- Coalesce duplicate in-flight queries for the same key where possible. + +## Benchmarking + +Benchmarks must cover: + +- Key generation throughput and allocation count (assert zero allocations). +- Cache lookup with a realistic number of entries. +- End-to-end query latency under a rebalance. + +Run benchmarks with `-benchmem` to verify the allocation-free guarantee: + +```bash +go test ./pkg/queryfrontend/... -bench . -benchmem -run '^$' +``` + +## Regression checklist + +Before merging a change that touches these paths, confirm: + +- [ ] Key generation still allocates zero bytes per call. +- [ ] Cache lookup remains O(1). +- [ ] End-to-end p99 latency is unchanged outside rebalances. diff --git a/docs/TESTING.md b/docs/TESTING.md new file mode 100644 index 0000000..5873ec6 --- /dev/null +++ b/docs/TESTING.md @@ -0,0 +1,34 @@ +# Testing Guide — Query Frontend + +This guide covers how to run and extend tests for the query frontend changes. + +## Running tests + +```bash +go vet ./pkg/queryfrontend/... +go test ./pkg/queryfrontend/... -count=1 +``` + +For race detection on the cache and invalidation paths: + +```bash +go test ./pkg/queryfrontend/... -race -run TestCache +``` + +## Test matrix + +| Area | What to assert | +|---|---| +| Cache key | Determinism for a fixed (tenant, query, epoch) triple | +| Cache key | A changed epoch yields a different key | +| Invalidation | In-flight query is not committed under a stale epoch | +| Concurrency | No data race between key generation and epoch reads | + +## Writing tests + +- Keep fixtures minimal; a single tenant/query pair is usually enough. +- Simulate a rebalance by bumping the epoch in the test and asserting the + observable behavior (new key, no stale commit). +- Use `-race` in CI for the cache path since it is concurrent by design. +- Avoid wall-clock time in assertions; the invalidation logic must be driven by + explicit epoch transitions, not timers. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md new file mode 100644 index 0000000..64a709e --- /dev/null +++ b/docs/TROUBLESHOOTING.md @@ -0,0 +1,70 @@ +# Troubleshooting Guide + +This guide helps diagnose and resolve common query-frontend issues. + +## Symptoms and remedies + +### Wrong query result served + +The cache returned a result for a different tenant or query. This indicates a +cache-key collision. + +**Cause:** the routing epoch is missing from the cache key. + +**Fix:** verify the key generator includes the tenant's active routing epoch, +and that the epoch is read atomically. + +### Partial result after a rebalance + +A query started before a rebalance returned a partial result afterwards. + +**Cause:** the mid-flight invalidation signal was not propagated. + +**Fix:** verify the invalidation signal is set for every query that spans a +rebalance boundary. + +### High CPU with no traffic increase + +Key generation is allocating on the hot path. + +**Cause:** the key builder regressed to `fmt.Sprintf` or repeated string +concatenation. + +**Fix:** restore the `strings.Builder` path and confirm zero allocations with +`go test -bench . -benchmem`. + +### Cache hit rate near zero + +The cache is not serving hits. + +**Cause:** the routing epoch is changing frequently, or the TTL is far too +short. + +**Fix:** investigate routing stability first, then adjust the TTL. + +### Non-deterministic keys + +The same query produces different keys across runs. + +**Cause:** wall-clock time or a random value entered the key. + +**Fix:** remove any time or randomness from the key; keys must depend only on +the tenant, query fingerprint, and epoch. + +## Diagnostic commands + +```bash +go vet ./pkg/queryfrontend/... +go test ./pkg/queryfrontend/... -count=1 +go test ./pkg/queryfrontend/... -race -run TestCache +go test ./pkg/queryfrontend/... -bench . -benchmem -run '^$' +``` + +## Escalation + +If a symptom cannot be explained by the causes above, capture the following +and escalate: + +- The cache key for a reproducing query (tenant, fingerprint, epoch). +- The routing epoch timeline around the incident. +- Benchmark output showing allocation counts.