Skip to content

Daemon background-work observability: CPU/mem self-report in comm.health + phase-logged background tasks #723

Description

@ohdearquant

Problem

The daemon accumulated ~514 CPU-minutes over a 12.5h window (cold start ~05:43Z to mid-afternoon), with multi-core bursts (300-800%) sustained long enough to corrupt co-tenant perf measurements on the host. Post-hoc investigation could not attribute the work from the daemon log: the log carries WARN-level checkpoint and FTS entries only. Background phases (ANN warm on cold start, embedding/index work, serving load) emit no start/end markers, so "what was the daemon doing between 06Z and 14Z" is unanswerable from the artifact that should answer it.

Asks

  1. Phase-level INFO logging for background tasks: ANN warm, index rebuild/backfill, and similar long-running phases log start (with corpus size), completion (with wall + CPU time), and cancellation. One line each — enough to attribute a CPU window after the fact.
  2. Daemon resource self-report in comm.health: extend the health snapshot with process-level CPU time (cumulative), RSS, and current background-phase name (if any). comm.health is already the sanctioned read surface for daemon state; perf-sensitive co-tenants (GPU flock holders) can poll it to know the box is dirty before measuring.
  3. QoS for warm-path work: consider lowering thread priority / niceness for cold-start warm tasks so they don't saturate the box; a cold-start warm competing with interactive serving and co-tenant measurement is the observed failure mode.

Non-goals

Scheduling/lockfile handshakes between seats are an operational convention, not daemon code. The daemon's job is to make its own state observable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions