diff --git a/README.md b/README.md index 63a7826..b87515a 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,7 @@ The file stores only the selected metrics, so select the limit of a usage bar to | Flag | Meaning | |------|---------| | `-o file` | output, `-` for stdout (the default); the extension picks the format | -| `-format f` | `png`, `svg`, `json`, `txt` or `ansi` | +| `-format f` | `png`, `svg`, `json`, `summary`, `txt` or `ansi` | | `-m glob`, `-f regex`, `-l matcher` | select metrics and series | | `-view v` | `auto` (details for one metric), `grid` or `detail` | | `-for d`, `-i d` | sample for `d`, every `-i` | @@ -154,6 +154,46 @@ The file stores only the selected metrics, so select the limit of a usage bar to Images are drawn in pure Go with an embedded DejaVu Sans Mono font; box drawing, bars and braille charts are geometry, so they join up at any scale. No browser, no server. +## For AI agents and scripts + +`-format summary` is the JSON to give a model or a script instead of the raw exposition: one line, +numbers in base units, labels as objects, `last/avg/min/max/trend` of every series over the +sampled window, and a ranked `notable` list of what is down, near a limit, answering 5xx, about to +expire, restarted, or moving. The layout is versioned (`"schema": "promtop.summary/v1"`) and does +not change with the web page, which is what `-format json` follows. + +```sh +promtop render -format summary -notable -for 1m 9100 # what is wrong, in a few hundred bytes +promtop render -format summary -list 9100 # which metrics there are, no values +promtop render -format summary -for 1m -m 'http_*' -l 'code=~5..' 9100 # dig into one thing +``` + +```json +{"schema":"promtop.summary/v1","source":"http://localhost:9100/metrics", + "window":{"from":"2026-10-29T14:40:49Z","to":"2026-10-29T14:41:49Z","samples":31,"interval_s":2}, + "totals":{"metrics":30,"series":58,"shown":26,"hidden":4}, + "notable":[ + {"metric":"demo_dependency_up","labels":{"dependency":"kafka"},"reason":"down","changes":2}, + {"metric":"demo_disk_avail_bytes","labels":{"mount":"/var/lib/demo"},"value":"used_ratio","reason":"near_limit","last":1}, + {"metric":"demo_http_requests_total","value":"rate","reason":"errors","last":0.5,"share":0.0076}]} +``` + +`-top n` keeps that many series per metric (5 by default, `0` for all), `-for` is what gives +trends (three samples or more), and `-state` with `-offline` reads the history `promtop record` +collected, so a cron job can ask a model about the last day. The detection is deterministic and +small — health flags, usage at 90%, 5xx counters, deadlines under 7 days, timestamps that moved, +and changes of 50% between the first and last third of the window that stand out of the noise +(Poisson for event counters, the spread of the samples otherwise; growing GC heaps are left out) — +the model does the interpreting. + +[`skills/analyzing-metrics-with-promtop`](skills/analyzing-metrics-with-promtop) is an +[Agent Skill](https://agentskills.io) that teaches an agent this workflow and the fields +(`reference.md` there documents the schema). For Claude Code: + +```sh +cp -r skills/analyzing-metrics-with-promtop ~/.claude/skills/ +``` + ## Install ```sh diff --git a/ROADMAP.md b/ROADMAP.md index 733cf2f..678b7e1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -18,6 +18,11 @@ off. Nothing here is promised; issues and pull requests are welcome. images show boxes where the terminal shows the text. - **Safe concurrent `record`.** Two writers at the same instant can lose one sample; a lock file would make it exact. +- **Agent summary: raw points.** A `-points` switch for `-format summary` that adds each series' + samples with timestamps, for questions the window statistics do not answer. +- **An MCP server** (`promtop mcp`) over the summary document, mainly to reach the history of a + running terminal or web session and to hand multimodal models the PNG of a chart; agents with a + shell already get everything else from `render -format summary`. - PDF output, and a size hint for chat apps (Telegram re-encodes photos above 1280 px). ## Dashboard layout diff --git a/internal/cli/cli_test.go b/internal/cli/cli_test.go index 7605660..07f0839 100644 --- a/internal/cli/cli_test.go +++ b/internal/cli/cli_test.go @@ -1,9 +1,11 @@ package cli import ( + "encoding/json" "image/png" "os" "path/filepath" + "strings" "testing" "github.com/igorrius/promtop/internal/history" @@ -34,6 +36,8 @@ func TestRenderCommand(t *testing.T) { {"-offline", fixture}, {"-view", "pie", "-o", filepath.Join(dir, "x.png"), fixture}, {"-l", "broken", fixture}, + {"-list", "-o", filepath.Join(dir, "x.png"), fixture}, + {"-format", "summary", "-list", "-notable", fixture}, } { if code := runRender(args); code != 2 { t.Errorf("render %v: exit %d, want 2", args, code) @@ -41,6 +45,35 @@ func TestRenderCommand(t *testing.T) { } } +func TestRenderSummary(t *testing.T) { + out := filepath.Join(t.TempDir(), "summary.json") + if code := runRender([]string{"-format", "summary", "-m", "go_*", "-o", out, fixture}); code != 0 { + t.Fatalf("exit %d", code) + } + b, err := os.ReadFile(out) + if err != nil { + t.Fatal(err) + } + var doc struct { + Schema string `json:"schema"` + Notable []json.RawMessage `json:"notable"` + Metrics []struct { + Name string `json:"name"` + } `json:"metrics"` + } + if err := json.Unmarshal(b, &doc); err != nil { + t.Fatalf("%v: %s", err, b) + } + if doc.Schema != "promtop.summary/v1" || doc.Notable == nil || len(doc.Metrics) == 0 { + t.Errorf("summary = %s", b) + } + for _, m := range doc.Metrics { + if !strings.HasPrefix(m.Name, "go_") { + t.Errorf("unselected metric %s", m.Name) + } + } +} + func TestRecordCommand(t *testing.T) { state := filepath.Join(t.TempDir(), "state") if code := runRecord([]string{fixture}); code != 2 { diff --git a/internal/cli/render.go b/internal/cli/render.go index 3604a3a..e942619 100644 --- a/internal/cli/render.go +++ b/internal/cli/render.go @@ -18,6 +18,7 @@ import ( "github.com/igorrius/promtop/internal/render" "github.com/igorrius/promtop/internal/scrape" "github.com/igorrius/promtop/internal/store" + "github.com/igorrius/promtop/internal/summary" "github.com/igorrius/promtop/internal/ui" "github.com/igorrius/promtop/internal/view" ) @@ -34,6 +35,13 @@ Examples: promtop render -m node_load1 -for 1m -o load.svg 9100 one metric: a detail chart promtop render -m 'temp_*' -state ~/.cache/temp.state -o temp.png sensors:9100 promtop render -format json -m 'battery_*' 9100 the data, for bots + promtop render -format summary -for 1m 9100 numbers and what changed, for agents + promtop render -format summary -list 9100 which metrics there are + +-format summary is a compact, versioned JSON document for AI agents and +scripts (schema promtop.summary/v1): values in base units, labels as +objects, last/avg/min/max/trend over the sampled window, and a "notable" +list of series that are down, near a limit, erroring or moving. Without -state or -for promtop scrapes twice, -i apart, so rates have a value but charts have only two points; -for samples longer, and -state draws the @@ -65,7 +73,7 @@ func runRender(args []string) int { sel := addSelectFlags(fs) var ( out = fs.String("o", "-", "output file, - for stdout") - format = fs.String("format", "", "png, svg, json, txt or ansi (default: from the -o extension, else png)") + format = fs.String("format", "", "png, svg, json, summary, txt or ansi (default: from the -o extension, else png)") style = fs.String("style", "terminal", "picture style: "+strings.Join(render.StyleNames(), ", ")) viewMode = fs.String("view", "auto", "grid, detail, or auto: details when one metric is selected") all = fs.Bool("a", false, "include metrics hidden in curated mode (idle, merged limits, runtime internals)") @@ -78,6 +86,9 @@ func runRender(args []string) int { width = fs.Int("width", 0, "width in cells (default: 90 for a grid, two columns; 100 for details)") height = fs.Int("height", 12, "chart height in cells of each detail view") scale = fs.Float64("scale", 2, "image scale: 1 means a 15px font") + top = fs.Int("top", 5, "-format summary: series per metric, 0 for all") + list = fs.Bool("list", false, "-format summary: only names, types, units and series counts") + notable = fs.Bool("notable", false, "-format summary: only the notable list") ) fs.Usage = func() { _, _ = fmt.Fprint(fs.Output(), renderUsage) @@ -95,10 +106,24 @@ func runRender(args []string) int { } } switch f { - case render.PNG, render.SVG, render.JSON, render.Text, render.ANSI: + case render.PNG, render.SVG, render.JSON, render.Text, render.ANSI, formatSummary: default: return usageErr(fmt.Errorf("unknown format %q", *format)) } + if f != formatSummary { + var only error + fs.Visit(func(fl *flag.Flag) { + if fl.Name == "top" || fl.Name == "list" || fl.Name == "notable" { + only = fmt.Errorf("-%s needs -format summary", fl.Name) + } + }) + if only != nil { + return usageErr(only) + } + } + if *list && *notable { + return usageErr(errors.New("-list and -notable do not combine")) + } if *out == "-" && f.Binary() && term.IsTerminal(os.Stdout.Fd()) { return usageErr(errors.New("refusing to write PNG to a terminal; use -o file.png or a pipe")) } @@ -125,42 +150,46 @@ func runRender(args []string) int { defer stop() st := store.New(*keep) interval := *sf.interval - var scrapes []history.Scrape + // raw is the history before the label matchers: what the summary's error + // share is taken from, so -l 'code=~5..' does not make it 100%. + var raw []history.Scrape + fresh := 0 if *statePath != "" { - if scrapes, err = history.Load(*statePath); err != nil { + if raw, err = history.Load(*statePath); err != nil { return fail(err) } - for i := range scrapes { - if scrapes[i].Families, err = selector.Apply(scrapes[i].Families); err != nil { - return usageErr(err) - } - } } if !*offline { - fresh, err := sampleScrapes(ctx, src, interval, *sample, len(scrapes) > 0) + // A list needs names, not rates: one scrape is enough. + got, err := sampleScrapes(ctx, src, interval, *sample, len(raw) > 0 || *list) switch { - case err != nil && len(scrapes) == 0: + case err != nil && len(raw) == 0: return fail(err) case err != nil: fmt.Fprintf(os.Stderr, "promtop: %v; drawing the stored history\n", err) } - for _, s := range fresh { + fresh = len(got) + for _, s := range got { // A history file stores only the selection; a one-off render keeps // every family, so usage panels still find their limits. if *statePath != "" { - s.Families, err = keepSelected(s.Families, names, selector) - } else { - s.Families, err = selector.Apply(s.Families) - } - if err != nil { - return usageErr(err) + if s.Families, err = keepSelected(s.Families, names, view.Selector{}); err != nil { + return usageErr(err) + } } - scrapes = history.Append(scrapes, s) + raw = history.Append(raw, s) } - if *statePath != "" && len(fresh) > 0 { - if err := history.Save(*statePath, scrapes, *keep); err != nil { - return fail(err) - } + } + scrapes := make([]history.Scrape, len(raw)) + for i, s := range raw { + if s.Families, err = selector.Apply(s.Families); err != nil { + return usageErr(err) + } + scrapes[i] = s + } + if *statePath != "" && fresh > 0 { + if err := history.Save(*statePath, scrapes, *keep); err != nil { + return fail(err) } } if len(scrapes) == 0 { @@ -219,12 +248,29 @@ func runRender(args []string) int { defer func() { _ = file.Close() }() w = file } - if err := render.Render(w, snap, *style, f, render.Options{Scale: *scale}); err != nil { + if f == formatSummary { + so := summary.Options{ + Source: src.String(), From: scrapes[0].At, To: snap.At, Interval: interval, Now: snap.Now, + Top: *top, List: *list, Notable: *notable, + } + if len(selector.Labels) > 0 { + so.Unfiltered = store.New(*keep) + history.Replay(so.Unfiltered, raw) + } + err = summary.Write(w, st, secs, so) + } else { + err = render.Render(w, snap, *style, f, render.Options{Scale: *scale}) + } + if err != nil { return fail(err) } return 0 } +// formatSummary is the agent document of package summary; it is not a +// picture of the dashboard, so it lives next to render's formats, not in them. +const formatSummary render.Format = "summary" + func runRecord(args []string) int { fs := flag.NewFlagSet("promtop record", flag.ContinueOnError) sf := addSourceFlags(fs, 2*time.Second) diff --git a/internal/summary/summary.go b/internal/summary/summary.go new file mode 100644 index 0000000..c5b0921 --- /dev/null +++ b/internal/summary/summary.go @@ -0,0 +1,704 @@ +// Package summary turns a dashboard into a compact JSON document for agents +// and scripts: numbers in base units, labels as objects, statistics over the +// sampled window instead of raw points, and a ranked list of what deserves a +// look. The layout is versioned (Schema) and does not follow the web page. +package summary + +import ( + "encoding/json" + "io" + "math" + "regexp" + "slices" + "sort" + "strconv" + "strings" + "time" + + "github.com/igorrius/promtop/internal/expo" + "github.com/igorrius/promtop/internal/store" + "github.com/igorrius/promtop/internal/view" +) + +// Schema names the document layout; it changes only incompatibly. +const Schema = "promtop.summary/v1" + +// Options shape the document. +type Options struct { + Source string + From, To time.Time // first and latest scrape of the window + Interval time.Duration + Now time.Time + Top int // series per metric, 0 for all + List bool // names, types and series counts only + Notable bool // only the notable list + // Unfiltered holds the same scrapes before label matchers, when there + // were any: an error share is of the metric's whole traffic. + Unfiltered *store.Store +} + +// Doc is the whole document. +type Doc struct { + Schema string `json:"schema"` + Source string `json:"source"` + Target string `json:"target,omitempty"` + Window Window `json:"window"` + Totals Totals `json:"totals"` + Notable *[]Note `json:"notable,omitempty"` // absent with List + Omitted int `json:"notable_omitted,omitempty"` + Metrics []Metric `json:"metrics,omitempty"` +} + +// Window is the sampled time range. +type Window struct { + From string `json:"from"` + To string `json:"to"` + Samples int `json:"samples"` + IntervalS float64 `json:"interval_s"` +} + +// Totals count what the target exposes and what the document shows. +type Totals struct { + Metrics int `json:"metrics"` // families in the exposition + Series int `json:"series"` + Shown int `json:"shown"` // metrics in this document + Hidden int `json:"hidden"` // idle, runtime noise or merged into a usage metric; -a shows them +} + +// Note is one thing worth a look. +type Note struct { + Metric string `json:"metric"` + Labels map[string]string `json:"labels,omitempty"` + Value string `json:"value,omitempty"` // what the numbers are: rate, p99, used_ratio... + Reason string `json:"reason"` + Last *Num `json:"last,omitempty"` + From *Num `json:"from,omitempty"` + To *Num `json:"to,omitempty"` + Change *Num `json:"change,omitempty"` + Share *Num `json:"share,omitempty"` // errors: of the metric's whole rate + InS *Num `json:"in_s,omitempty"` + Count int `json:"changes,omitempty"` +} + +// Reasons, most urgent first. +const ( + ReasonDown = "down" // a health flag is 0 + ReasonNearLimit = "near_limit" // usage at 90% or more + ReasonErrors = "errors" // 5xx responses are being counted, all series of a metric together + ReasonExpiresSoon = "expires_soon" // a timestamp less than 7 days ahead + ReasonFlapped = "flapped" // a health flag changed during the window + ReasonChanged = "changed" // a timestamp moved: restart, deploy, reload + ReasonRising = "rising" + ReasonFalling = "falling" +) + +var urgency = map[string]int{ + ReasonDown: 0, ReasonNearLimit: 1, ReasonErrors: 2, ReasonExpiresSoon: 3, + ReasonFlapped: 4, ReasonChanged: 5, ReasonRising: 6, ReasonFalling: 6, +} + +// Metric is one metric family. +type Metric struct { + Name string `json:"name"` + Type string `json:"type"` // exposition type + Kind string `json:"kind"` // how promtop reads it: gauge, rate, histogram, usage, state... + Unit string `json:"unit,omitempty"` + Value string `json:"value,omitempty"` // what last/avg/min/max of a series describe + Help string `json:"help,omitempty"` + Labels map[string]string `json:"labels,omitempty"` // shared by every series + Limit string `json:"limit,omitempty"` // usage: the metric holding the maximum + Count int `json:"series_count"` + Sum *Stats `json:"sum,omitempty"` // rates: all series added up + Series []Series `json:"series,omitempty"` + Omitted int `json:"series_omitted,omitempty"` +} + +// Stats describe one value over the window. +type Stats struct { + Last *Num `json:"last,omitempty"` + Avg *Num `json:"avg,omitempty"` + Min *Num `json:"min,omitempty"` + Max *Num `json:"max,omitempty"` + Trend string `json:"trend,omitempty"` // rising, falling or flat; needs 3 samples + Change *Num `json:"change,omitempty"` // relative, first third of the window to the last + from float64 // means of the first and the last third + to float64 + spread float64 // sum of both thirds' sample variances + third int // samples in a third +} + +func (st Stats) avgOr0() float64 { + if st.Avg == nil { + return 0 + } + return float64(*st.Avg) +} + +// Series is one label combination. +type Series struct { + Labels map[string]string `json:"labels,omitempty"` + Stats + Total *Num `json:"total,omitempty"` // counter: lifetime value + Rate *Num `json:"rate,omitempty"` // histogram, summary: observations per second + Mean *Num `json:"mean,omitempty"` // histogram, summary: average observation + Quantiles map[string]Num `json:"quantiles,omitempty"` + Used *Num `json:"used,omitempty"` + Limit *Num `json:"limit,omitempty"` + Up *bool `json:"up,omitempty"` // health flag + Active *bool `json:"active,omitempty"` // stateset member + Changes int `json:"changes,omitempty"` + Time string `json:"time,omitempty"` // timestamp metrics + AgeS *Num `json:"age_s,omitempty"` // a past timestamp: seconds ago + InS *Num `json:"in_s,omitempty"` // a future one: seconds ahead +} + +// Num is a float written with six significant digits. +type Num float64 + +// MarshalJSON writes the number compactly. +func (n Num) MarshalJSON() ([]byte, error) { + return strconv.AppendFloat(nil, float64(n), 'g', 6, 64), nil +} + +func num(v float64) *Num { + if math.IsNaN(v) || math.IsInf(v, 0) { + return nil + } + n := Num(v) + return &n +} + +// Write encodes the document for the store and its dashboard sections. +func Write(w io.Writer, st *store.Store, secs []*view.Section, o Options) error { + // Compact: the reader is a program or a model paying per token. + return json.NewEncoder(w).Encode(Build(st, secs, o)) +} + +// Build makes the document. +func Build(st *store.Store, secs []*view.Section, o Options) Doc { + d := Doc{ + Schema: Schema, Source: o.Source, Target: view.Target(st), + Window: Window{ + From: stamp(o.From), To: stamp(o.To), Samples: min(st.Scrapes, st.MaxPoints), + IntervalS: o.Interval.Seconds(), + }, + } + for _, f := range st.Families { + if n := len(f.Live()); n > 0 { + d.Totals.Metrics++ + d.Totals.Series += n + } + } + var notes []Note + for _, sec := range secs { + d.Totals.Hidden += sec.Hidden + for _, p := range expand(st, sec.Panels) { + d.Totals.Shown++ + if !o.List { + notes = append(notes, notable(p, o)...) + } + if !o.Notable { + d.Metrics = append(d.Metrics, metric(p, o)) + } + } + } + sort.SliceStable(notes, func(i, j int) bool { + a, b := notes[i], notes[j] + if urgency[a.Reason] != urgency[b.Reason] { + return urgency[a.Reason] < urgency[b.Reason] + } + return magnitude(a) > magnitude(b) + }) + if len(notes) > maxNotes { + d.Omitted = len(notes) - maxNotes + notes = notes[:maxNotes] + } + if !o.List { + notes = append([]Note{}, notes...) + d.Notable = ¬es + } + return d +} + +const maxNotes = 20 + +func magnitude(n Note) float64 { + if n.Change == nil { + return math.Inf(1) // appeared from zero + } + return math.Abs(float64(*n.Change)) +} + +func stamp(t time.Time) string { + if t.IsZero() { + return "" + } + return t.UTC().Format(time.RFC3339) +} + +// expand replaces group panels, which pack single-series metrics of a +// crowded section for the screen, by their members: an agent wants names. +func expand(st *store.Store, panels []*view.Panel) []*view.Panel { + var out []*view.Panel + for _, p := range panels { + if !p.Group { + out = append(out, p) + continue + } + for _, s := range p.Family.Series { + if f := st.Get(s.Key); f != nil { + if m := view.Classify(f, st); m != nil { + out = append(out, m) + } + } + } + } + return out +} + +func metric(p *view.Panel, o Options) Metric { + f := p.Family + live := p.Rows() + m := Metric{ + Name: f.Name, Type: string(f.Type), Kind: p.Kind.String(), Unit: unit(p), + Value: valueName(p), Help: f.Help, Labels: labelMap(f.CommonLabels), Count: len(live), + } + if p.Limit != nil { + m.Limit = p.Limit.Name + } + if o.List { + return m + } + if p.Kind == view.KindRate && len(live) > 1 { + s := stats(sumRings(live, scaleOf(p))) + m.Sum = &s + } + if o.Top > 0 && len(live) > o.Top { + m.Omitted = len(live) - o.Top + live = live[:o.Top] + } + for _, s := range live { + m.Series = append(m.Series, series(p, s, o.Now)) + } + return m +} + +func series(p *view.Panel, s *store.Series, now time.Time) Series { + out := Series{Labels: labelMap(view.VisibleLabels(p.Family, s))} + k := scaleOf(p) + switch p.Kind { + case view.KindInfo: + return out + case view.KindTimestamp: + t := p.Time(s) + out.Time = stamp(t) + if d := now.Sub(t).Seconds(); d >= 0 { + out.AgeS = num(d) + } else { + out.InS = num(-d) + } + return out + case view.KindState: + on := view.StateUp(s) + if p.Family.Type == expo.StateSet { + out.Active = &on + } else { + out.Up = &on + } + out.Changes = transitions(values(s.Primary())) + return out + case view.KindUsage: + used, lim := p.UsageValues(s) + out.Used, out.Limit = num(used*k), num(lim*k) + case view.KindHistogram, view.KindSummary: + out.Rate = num(s.Line("rate").Last()) + out.Mean = num(s.Line("avg").Last() * k) + out.Quantiles = map[string]Num{} + for _, l := range s.Lines { + if l.Name == "rate" || l.Name == "avg" { + continue + } + if v := num(l.Ring.Last() * k); v != nil { + out.Quantiles[l.Name] = *v + } + } + case view.KindRate: + out.Total = num(s.Raw * k) + } + out.Stats = stats(history(p, s)) + return out +} + +// history is the value of s that the panel plots, in base units, oldest +// first: the rate, a tail quantile, the used fraction or the gauge value. +func history(p *view.Panel, s *store.Series) []float64 { + var vals []float64 + switch p.Kind { + case view.KindUsage: + return usageRing(p, s) + case view.KindHistogram, view.KindSummary: + vals = slices.Clone(values(view.Ring(s, p.RowLine()))) + if p.Kind == view.KindHistogram { + // Before the first rate a histogram's quantiles describe its whole + // lifetime, not the interval; they would fake a trend. + rate := values(s.Line("rate")) + for i := range vals { + if j := len(rate) - len(vals) + i; j < 0 || math.IsNaN(rate[j]) { + vals[i] = math.NaN() + } + } + } + default: + vals = slices.Clone(values(s.Primary())) + } + if k := scaleOf(p); k != 1 { + for i := range vals { + vals[i] *= k + } + } + return vals +} + +func scaleOf(p *view.Panel) float64 { + if p.Scale == 0 { + return 1 + } + return p.Scale +} + +// notable lists the series of p that deserve a look. +func notable(p *view.Panel, o Options) []Note { + if p.Kind == view.KindState && p.Family.Type == expo.StateSet { + return stateSetNotes(p) + } + var out []Note + for _, s := range p.Family.Live() { + note := func(reason string) Note { + return Note{Metric: p.Name(), Labels: labelMap(view.VisibleLabels(p.Family, s)), Value: valueName(p), Reason: reason} + } + switch p.Kind { + case view.KindInfo: + continue + case view.KindState: + n := transitions(values(s.Primary())) + switch { + case !view.StateUp(s): + nt := note(ReasonDown) + nt.Count = n + out = append(out, nt) + case n > 0: + nt := note(ReasonFlapped) + nt.Count = n + out = append(out, nt) + } + continue + case view.KindTimestamp: + if d := p.Time(s).Sub(o.Now); d > 0 && d < 7*24*time.Hour && expiry(p.Name()) { + nt := note(ReasonExpiresSoon) + nt.InS = num(d.Seconds()) + out = append(out, nt) + } else if transitions(values(s.Primary())) > 0 { + out = append(out, note(ReasonChanged)) + } + continue + } + + st := stats(history(p, s)) + switch { + case p.Kind == view.KindUsage && p.Usage(s) >= nearLimit: + nt := note(ReasonNearLimit) + nt.Last = num(p.Usage(s)) + out = append(out, nt) + case p.Kind == view.KindRate && serverError(s): + // Counted once per metric below. + case !gcHeap.MatchString(p.Name()) && moved(p, s, st, o.Interval): + nt := note(ReasonRising) + if st.Trend == ReasonFalling { + nt.Reason = ReasonFalling + } + nt.Last, nt.From, nt.To, nt.Change = st.Last, num(st.from), num(st.to), st.Change + out = append(out, nt) + } + } + if p.Kind == view.KindRate { + f := p.Family + if o.Unfiltered != nil && o.Unfiltered.Get(f.Name) != nil { + f = o.Unfiltered.Get(f.Name) + } + if errs, all := errorRate(f.Live()); errs > 0 { + out = append(out, Note{ + Metric: p.Name(), Value: "rate", Reason: ReasonErrors, + Last: num(errs * scaleOf(p)), Share: num(errs / all), + }) + } + } + return out +} + +// stateSetNotes reports a stateset that switched during the window, once, +// labelled with the state it is in now. +func stateSetNotes(p *view.Panel) []Note { + n, active := 0, map[string]string(nil) + for _, s := range p.Family.Live() { + n = max(n, transitions(values(s.Primary()))) + if view.StateUp(s) { + active = labelMap(view.VisibleLabels(p.Family, s)) + } + } + if n == 0 { + return nil + } + return []Note{{Metric: p.Name(), Labels: active, Reason: ReasonFlapped, Count: n}} +} + +// errorRate adds up the latest rates of 5xx series and of all series. +func errorRate(live []*store.Series) (errs, all float64) { + for _, s := range live { + v := s.Primary().Last() + if math.IsNaN(v) { + continue + } + all += v + if serverError(s) { + errs += v + } + } + return errs, all +} + +// gcHeap matches garbage-collected heap gauges: they climb until the next +// collection, often for longer than the window, so a rise is no news. A +// leak shows in the resident memory, which keeps its trend notes. +var gcHeap = regexp.MustCompile(`(?i)heap|^go_memstats_alloc_bytes$|^go_memory_used|^jvm_memory_(bytes_)?used`) + +const ( + nearLimit = 0.9 + bigChange = 0.5 // relative change between the window's thirds that is notable + minEvents = 20 // observations per third before a quantile's trend means anything +) + +// moved reports whether the trend of st is big enough to mention, and more +// than the noise of counting few events. +func moved(p *view.Panel, s *store.Series, st Stats, interval time.Duration) bool { + if st.Trend != ReasonRising && st.Trend != ReasonFalling { + return false + } + if st.Change != nil && math.Abs(float64(*st.Change)) < bigChange { + return false + } + span := float64(st.third) * interval.Seconds() // seconds in a third of the window + if p.Kind == view.KindRate && p.Unit == view.UnitNone { + // Event counts are Poisson: a third with c events varies by about √c. + c1, c2 := st.from*span, st.to*span + return math.Abs(c2-c1) > 3*math.Sqrt(c1+c2) + } + if (p.Kind == view.KindHistogram || p.Kind == view.KindSummary) && + stats(values(s.Line("rate"))).avgOr0()*span < minEvents { + return false + } + // Anything else must move more than it wobbles within the thirds (a + // Welch test at 3σ), so a GC heap's sawtooth is not a trend. That needs + // two samples per third. + return st.third >= 2 && math.Abs(st.to-st.from) > 3*math.Sqrt(st.spread/float64(st.third)) +} + +// expiry reports whether a timestamp metric names a deadline (a certificate's +// notAfter) rather than a past event. +func expiry(name string) bool { + n := strings.ToLower(name) + return strings.Contains(n, "expir") || strings.Contains(n, "not_after") || strings.Contains(n, "notafter") || strings.Contains(n, "deadline") +} + +var statusLabels = []string{"code", "status", "status_code", "http_status_code", "http_response_status_code"} + +func serverError(s *store.Series) bool { + for _, l := range s.Labels { + for _, n := range statusLabels { + if l.Name == n && len(l.Value) == 3 && l.Value[0] == '5' { + return true + } + } + } + return false +} + +// stats describes vals, ignoring gaps. +func stats(vals []float64) Stats { + var xs []float64 + for _, v := range vals { + if !math.IsNaN(v) && !math.IsInf(v, 0) { + xs = append(xs, v) + } + } + if len(xs) == 0 { + return Stats{} + } + lo, hi, sum := xs[0], xs[0], 0.0 + for _, v := range xs { + lo, hi, sum = min(lo, v), max(hi, v), sum+v + } + st := Stats{Last: num(xs[len(xs)-1]), Avg: num(sum / float64(len(xs))), Min: num(lo), Max: num(hi)} + if len(xs) < 3 { + return st + } + third := max(1, len(xs)/3) + st.third = third + st.from, st.to = mean(xs[:third]), mean(xs[len(xs)-third:]) + st.spread = variance(xs[:third]) + variance(xs[len(xs)-third:]) + switch st.from { + case st.to: + st.Trend = "flat" + case 0: + st.Trend = ReasonRising + if st.to < 0 { + st.Trend = ReasonFalling + } + default: + c := (st.to - st.from) / math.Abs(st.from) + st.Change = num(c) + switch { + case c >= 0.1: + st.Trend = ReasonRising + case c <= -0.1: + st.Trend = ReasonFalling + default: + st.Trend = "flat" + } + } + return st +} + +// variance is the sample variance of xs, 0 for fewer than two values. +func variance(xs []float64) float64 { + if len(xs) < 2 { + return 0 + } + m, s := mean(xs), 0.0 + for _, v := range xs { + s += (v - m) * (v - m) + } + return s / float64(len(xs)-1) +} + +func mean(xs []float64) float64 { + s := 0.0 + for _, v := range xs { + s += v + } + return s / float64(len(xs)) +} + +// transitions counts value changes, ignoring gaps. +func transitions(vals []float64) int { + n, prev, have := 0, 0.0, false + for _, v := range vals { + if math.IsNaN(v) { + continue + } + if have && v != prev { + n++ + } + prev, have = v, true + } + return n +} + +// sumRings adds the series' primary histories, aligned at the newest value. +func sumRings(ss []*store.Series, k float64) []float64 { + n := 0 + for _, s := range ss { + n = max(n, len(values(s.Primary()))) + } + out := make([]float64, n) + for i := range out { + out[i] = math.NaN() + } + for _, s := range ss { + vals := values(s.Primary()) + for i, v := range vals { + j := n - len(vals) + i + if math.IsNaN(v) { + continue + } + if math.IsNaN(out[j]) { + out[j] = 0 + } + out[j] += v * k + } + } + return out +} + +// usageRing is the used fraction of s over the window, against the latest +// limit: limits rarely move, and their history is not kept per usage row. +func usageRing(p *view.Panel, s *store.Series) []float64 { + _, lim := p.UsageValues(s) + if lim <= 0 { + return nil + } + vals := values(s.Primary()) + out := make([]float64, len(vals)) + for i, v := range vals { + if p.LimitInvert { + v = lim - v + } + out[i] = v / lim + } + return out +} + +// values is the history of r, nil for a missing line. +func values(r *store.Ring) []float64 { + if r == nil { + return nil + } + return r.Values() +} + +func valueName(p *view.Panel) string { + switch p.Kind { + case view.KindRate: + return "rate" + case view.KindHistogram, view.KindSummary: + return p.RowLine() + case view.KindUsage: + return "used_ratio" + case view.KindGauge: + return "value" + } + return "" +} + +func unit(p *view.Panel) string { + var u string + switch p.Unit { + case view.UnitBytes: + u = "bytes" + case view.UnitSeconds: + u = "seconds" + case view.UnitRatio: + u = "ratio" + case view.UnitPercent: + u = "percent" + case view.UnitOther: + u = p.Sym + } + if p.Kind == view.KindRate { + if u == "" { + return "1/s" + } + return u + "/s" + } + return u +} + +func labelMap(ls []expo.Label) map[string]string { + if len(ls) == 0 { + return nil + } + m := make(map[string]string, len(ls)) + for _, l := range ls { + m[l.Name] = l.Value + } + return m +} diff --git a/internal/summary/summary_test.go b/internal/summary/summary_test.go new file mode 100644 index 0000000..9c9d3ee --- /dev/null +++ b/internal/summary/summary_test.go @@ -0,0 +1,313 @@ +package summary + +import ( + "bytes" + "encoding/json" + "fmt" + "math" + "strings" + "testing" + "time" + + "github.com/igorrius/promtop/internal/expo" + "github.com/igorrius/promtop/internal/store" + "github.com/igorrius/promtop/internal/view" +) + +var t0 = time.Unix(1_800_000_000, 0) + +// ingest feeds n scrapes 10s apart; gen returns the exposition of scrape i. +func ingest(t *testing.T, n int, gen func(i int) string) *store.Store { + t.Helper() + st := store.New(100) + for i := range n { + fams, err := expo.Parse(strings.NewReader(gen(i))) + if err != nil { + t.Fatal(err) + } + st.Ingest(fams, t0.Add(time.Duration(i)*10*time.Second)) + } + return st +} + +func build(st *store.Store, o Options) Doc { + o.Interval = 10 * time.Second + o.Now = t0.Add(time.Duration(st.Scrapes-1) * 10 * time.Second) + o.From, o.To = t0, o.Now + return Build(st, view.Build(st, view.Options{}), o) +} + +func find(d Doc, name string) *Metric { + for i := range d.Metrics { + if d.Metrics[i].Name == name { + return &d.Metrics[i] + } + } + return nil +} + +func notes(d Doc, metric string) []Note { + var out []Note + for _, n := range *d.Notable { + if n.Metric == metric { + out = append(out, n) + } + } + return out +} + +func TestTrends(t *testing.T) { + // app_busy_total doubles its rate from 100/s to 200/s halfway; app_rare_total + // counts a handful of events, so its "trend" is noise. + busy, rare := 0.0, 0.0 + st := ingest(t, 9, func(i int) string { + if i > 0 { + busy += 1000 + if i >= 5 { + busy += 1000 + } + rare += float64(i % 2) + } + return fmt.Sprintf("# TYPE app_busy_total counter\napp_busy_total %g\n# TYPE app_rare_total counter\napp_rare_total %g\n", busy, rare) + }) + d := build(st, Options{}) + + n := notes(d, "app_busy_total") + if len(n) != 1 || n[0].Reason != ReasonRising || n[0].Change == nil || *n[0].Change < 0.5 { + t.Fatalf("busy notes = %+v", n) + } + if n := notes(d, "app_rare_total"); len(n) != 0 { + t.Errorf("a few events should not make a trend: %+v", n) + } + m := find(d, "app_busy_total") + if m == nil || m.Unit != "1/s" || m.Value != "rate" || len(m.Series) != 1 { + t.Fatalf("metric = %+v", m) + } + s := m.Series[0] + if *s.Last != 200 || *s.Min != 100 || float64(*s.Total) != busy || s.Trend != ReasonRising { + t.Errorf("series = %+v", s) + } +} + +func TestGaugeTrends(t *testing.T) { + // app_queue climbs steadily; app_jitter jumps around as much as it moves; + // a Go heap climbs until the next GC. + queue := []int{10, 11, 12, 13, 14, 15, 16, 17, 18} + jitter := []int{400, 1600, 500, 1500, 450, 1650, 1700, 600, 1800} + st := ingest(t, 9, func(i int) string { + return fmt.Sprintf("# TYPE app_queue gauge\napp_queue %d\n# TYPE app_jitter gauge\napp_jitter %d\n"+ + "# TYPE go_memstats_heap_alloc_bytes gauge\ngo_memstats_heap_alloc_bytes %d\n", queue[i], jitter[i], 1e6*queue[i]) + }) + d := build(st, Options{}) + if n := notes(d, "app_queue"); len(n) != 1 || n[0].Reason != ReasonRising { + t.Errorf("queue = %+v", n) + } + if n := notes(d, "app_jitter"); len(n) != 0 { + t.Errorf("a sawtooth is not a trend: %+v", n) + } + if n := notes(d, "go_memstats_heap_alloc_bytes"); len(n) != 0 { + t.Errorf("a growing heap is no news: %+v", n) + } + if s := find(d, "app_jitter").Series[0]; s.Trend != ReasonRising { + t.Errorf("the series still shows its trend: %+v", s) + } +} + +func TestErrorShareUnfiltered(t *testing.T) { + all := ingest(t, 3, func(i int) string { + return fmt.Sprintf("# TYPE req_total counter\nreq_total{code=\"200\"} %d\nreq_total{code=\"500\"} %d\n", 300*i, 100*i) + }) + only5xx := ingest(t, 3, func(i int) string { + return fmt.Sprintf("# TYPE req_total counter\nreq_total{code=\"500\"} %d\n", 100*i) + }) + d := build(only5xx, Options{Unfiltered: all}) + if n := notes(d, "req_total"); len(n) != 1 || *n[0].Share != 0.25 || *n[0].Last != 10 { + t.Errorf("share = %+v", n) + } +} + +func TestErrorsAndStates(t *testing.T) { + st := ingest(t, 4, func(i int) string { + up := 1 + if i == 2 { + up = 0 // a flap + } + return fmt.Sprintf(`# TYPE http_requests_total counter +http_requests_total{code="200",path="/a"} %d +http_requests_total{code="503",path="/a"} %d +# TYPE dep_up gauge +dep_up{dep="db"} 0 +dep_up{dep="cache"} %d +# TYPE breaker stateset +breaker{breaker="closed"} %d +breaker{breaker="open"} %d +`, 900*i, 100*i, up, up, 1-up) + }) + d := build(st, Options{}) + + n := notes(d, "http_requests_total") + if len(n) != 1 || n[0].Reason != ReasonErrors || *n[0].Last != 10 || math.Abs(float64(*n[0].Share)-0.1) > 1e-9 { + t.Fatalf("errors = %+v", n) + } + got := map[string]string{} + for _, n := range notes(d, "dep_up") { + got[n.Labels["dep"]] = n.Reason + } + if got["db"] != ReasonDown || got["cache"] != ReasonFlapped { + t.Errorf("states = %v", got) + } + if n := notes(d, "breaker"); len(n) != 1 || n[0].Reason != ReasonFlapped || n[0].Labels["breaker"] != "closed" || n[0].Count != 2 { + t.Errorf("stateset = %+v", n) + } + if (*d.Notable)[0].Reason != ReasonDown { + t.Errorf("down should come first: %+v", *d.Notable) + } + m := find(d, "dep_up") + for _, s := range m.Series { + if s.Up == nil || *s.Up != (s.Labels["dep"] == "cache") { + t.Errorf("state series = %+v", s) + } + } +} + +func TestUsageAndUnits(t *testing.T) { + st := ingest(t, 3, func(i int) string { + return fmt.Sprintf(`# TYPE fs_avail_bytes gauge +fs_avail_bytes{mount="/data"} %d +fs_avail_bytes{mount="/"} 600 +# TYPE fs_size_bytes gauge +fs_size_bytes{mount="/data"} 1000 +fs_size_bytes{mount="/"} 1000 +# TYPE job_wait_milliseconds gauge +job_wait_milliseconds{instance="a",queue="x"} 250 +job_wait_milliseconds{instance="a",queue="y"} 100 +`, 50-10*i) + }) + d := build(st, Options{}) + + n := notes(d, "fs_avail_bytes") + if len(n) != 1 || n[0].Reason != ReasonNearLimit || n[0].Labels["mount"] != "/data" || *n[0].Last != 0.97 { + t.Fatalf("usage notes = %+v", n) + } + m := find(d, "fs_avail_bytes") + if m.Kind != "usage" || m.Limit != "fs_size_bytes" || m.Value != "used_ratio" { + t.Fatalf("usage metric = %+v", m) + } + if s := m.Series[0]; s.Labels["mount"] != "/data" || *s.Used != 970 || *s.Limit != 1000 || *s.Last != 0.97 { + t.Errorf("fullest mount first: %+v", s) + } + if find(d, "fs_size_bytes") != nil { + t.Error("the limit is part of the usage metric") + } + + w := find(d, "job_wait_milliseconds") + if w.Unit != "seconds" || *w.Series[0].Last != 0.25 { + t.Errorf("milliseconds should be converted: %+v", w) + } + if w.Labels["instance"] != "a" || len(w.Series[0].Labels) != 1 { + t.Errorf("shared labels belong to the metric: %+v", w) + } +} + +func TestHistogramIgnoresLifetime(t *testing.T) { + // A slow past (every lifetime observation above 1s) and a fast present: the + // first scrape's lifetime quantiles must not read as a falling trend. + st := ingest(t, 6, func(i int) string { + fast := 1000 * i + return fmt.Sprintf(`# TYPE rpc_seconds histogram +rpc_seconds_bucket{le="0.1"} %d +rpc_seconds_bucket{le="1"} %d +rpc_seconds_bucket{le="10"} %d +rpc_seconds_bucket{le="+Inf"} %d +rpc_seconds_sum 1 +rpc_seconds_count %d +`, fast, fast, fast+5000, fast+5000, fast+5000) + }) + d := build(st, Options{}) + if n := notes(d, "rpc_seconds"); len(n) != 0 { + t.Errorf("notes = %+v", n) + } + s := find(d, "rpc_seconds").Series[0] + if s.Trend != "flat" || *s.Max > 0.1 || *s.Rate != 100 || s.Quantiles["p50"] > 0.1 { + t.Errorf("series = %+v", s) + } +} + +func TestTimestamps(t *testing.T) { + st := ingest(t, 3, func(i int) string { + return fmt.Sprintf(`# TYPE tls_cert_expiry_timestamp_seconds gauge +tls_cert_expiry_timestamp_seconds %d +# TYPE process_start_time_seconds gauge +process_start_time_seconds %d +`, t0.Add(48*time.Hour).Unix(), t0.Unix()+int64(10*(i/2))) + }) + d := build(st, Options{}) + if n := notes(d, "tls_cert_expiry_timestamp_seconds"); len(n) != 1 || n[0].Reason != ReasonExpiresSoon || *n[0].InS != 48*3600-20 { + t.Errorf("expiry = %+v", n) + } + if n := notes(d, "process_start_time_seconds"); len(n) != 1 || n[0].Reason != ReasonChanged { + t.Errorf("restart = %+v", n) + } + s := find(d, "process_start_time_seconds").Series[0] + if s.Time != "2027-01-15T08:00:10Z" || *s.AgeS != 10 { + t.Errorf("time = %+v", s) + } +} + +func TestShape(t *testing.T) { + gen := func(i int) string { + var b strings.Builder + b.WriteString("# TYPE req_total counter\n") + for p := range 8 { + fmt.Fprintf(&b, "req_total{path=\"/%d\"} %d\n", p, p*i) + } + // Enough single-series gauges for the dashboard to group them. + for _, n := range []string{"a", "b", "c", "d", "e", "f"} { + fmt.Fprintf(&b, "# TYPE mem_%s_bytes gauge\nmem_%s_bytes 1\n", n, n) + } + return b.String() + } + st := ingest(t, 2, gen) + + d := build(st, Options{Top: 3}) + m := find(d, "req_total") + if len(m.Series) != 3 || m.Omitted != 5 || m.Count != 8 || m.Series[0].Labels["path"] != "/7" { + t.Errorf("top = %+v", m) + } + if find(d, "mem_a_bytes") == nil || d.Totals.Shown != 7 { + t.Errorf("groups should be expanded into metrics: %+v", d.Totals) + } + + var buf bytes.Buffer + if err := Write(&buf, st, view.Build(st, view.Options{}), Options{List: true}); err != nil { + t.Fatal(err) + } + var raw map[string]any + if err := json.Unmarshal(buf.Bytes(), &raw); err != nil { + t.Fatal(err) + } + if raw["schema"] != Schema || raw["notable"] != nil { + t.Errorf("list doc = %s", buf.String()) + } + if strings.Contains(buf.String(), `"series":[`) || strings.Count(buf.String(), "\n") != 1 { + t.Errorf("a list has no series and is one line: %s", buf.String()) + } + + buf.Reset() + if err := Write(&buf, st, view.Build(st, view.Options{}), Options{Notable: true}); err != nil { + t.Fatal(err) + } + if !strings.Contains(buf.String(), `"notable":[]`) || strings.Contains(buf.String(), `"metrics":[`) { + t.Errorf("notable doc = %s", buf.String()) + } +} + +func TestNum(t *testing.T) { + b, _ := json.Marshal(struct { + A *Num `json:"a,omitempty"` + B *Num `json:"b,omitempty"` + }{num(math.NaN()), num(1.0 / 3)}) + if string(b) != `{"b":0.333333}` { + t.Errorf("got %s", b) + } +} diff --git a/skills/analyzing-metrics-with-promtop/SKILL.md b/skills/analyzing-metrics-with-promtop/SKILL.md new file mode 100644 index 0000000..8ca2dd8 --- /dev/null +++ b/skills/analyzing-metrics-with-promtop/SKILL.md @@ -0,0 +1,73 @@ +--- +name: analyzing-metrics-with-promtop +description: Use when asked to look at, explain, check or troubleshoot a service through its Prometheus /metrics endpoint (exporter, node_exporter, client_golang, OpenTelemetry Prometheus exporter) or a saved .prom exposition, and promtop is installed - "what's wrong with the service", "is latency up", "why are there errors", "how full is the disk", a health report or a daily summary from metrics. +--- + +# Analyzing metrics with promtop + +## Overview + +Raw `/metrics` is a poor input for a model: thousands of lines, counters that only make sense as +rates between two scrapes, histogram buckets that need arithmetic. `promtop render -format summary` +does the arithmetic and prints a compact JSON document: rates, quantiles, used fractions, trends +over a sampled window, and a ranked `notable` list. Read that, not the exposition. + +## Workflow + +1. **What is wrong?** Sample a window and read only the notable list (`metrics` is left out): + `promtop render -format summary -notable -for 1m ` + When the question names a metric ("how bad is latency"), skip to step 3 and select it as well: + the full document has `notable` too, and each `-for` run costs its whole duration. +2. **What is there?** Names, types, units and series counts, no values (one scrape, fast): + `promtop render -format summary -list ` +3. **Dig in.** Select the metrics the question is about (`-m` repeats; any match is kept): + `promtop render -format summary -for 1m -m 'http_request_duration_seconds' -m 'node_filesystem_*' ` +4. **Show people** the same selection as a picture when a human will read the answer: + `promtop render -for 1m -m 'http_request_duration_seconds' -o latency.png ` + +``: `9100` = `localhost:9100/metrics`, `host:9100`, a full URL, or `./dump.prom`. +Add `-H 'Authorization: Bearer …'` for protected endpoints. + +## Quick reference + +| Flag | Use | +|---|---| +| `-for 1m` | sample a window (`30s`, `5m`); **needed for `trend`** (3+ samples, 6+ for trend notes). Default is 2 scrapes: rates, no trends | +| `-i 5s` | sample interval (default 2s) | +| `-m glob`, `-f regex` | select metrics; `-m` repeats (`-m 'http_*' -m 'db_*'`), no comma lists | +| `-l matcher` | keep series matching (`mount=/data`, `'code=~5..'`, `dev!=lo`) **in every selected metric**: metrics without that label disappear. One `-l` query per question | +| `-top n` | series per metric (default 5, `0` = all); cut series are counted in `series_omitted` | +| `-a` | include hidden metrics (idle, Go/process runtime noise, limits merged into usage); `-a -list` names them | +| `-state file -offline` | read history collected by `promtop record` from cron: hours or days, no scraping | + +## Reading the document + +- `notable[]` is sorted most urgent first: `down`, `near_limit` (≥90%), `errors` (5xx, whole metric, + `share` of its traffic), `expires_soon` (<7 days), `flapped`, `changed` (restart, deploy), + `rising`/`falling` (≥50% between the first and last third of the window, beyond counting noise). +- Numbers are in base units named by the metric's `unit`: seconds, bytes, `1/s`, `bytes/s`. + `seconds/s` is CPU cores. `used_ratio` is 0..1 and always means *used*, also for + `*_avail_bytes`/`*_free_*` metrics (promtop inverts them against their size): `1` is full. +- Each series has `last/avg/min/max` of the metric's `value` (`rate`, `p99`, `used_ratio`, `value`) + and `trend` with `change` (relative: `1.5` = +150%). Histograms add `rate`, `mean`, `quantiles`. +- `labels` on a metric are shared by all its series; a series lists only its own. + +Full field list: `reference.md` next to this file. + +## Common mistakes + +- **Curling `/metrics` and reading it yourself.** Counter totals and buckets mislead; use the summary. +- **Reporting "no trend" from a 2-scrape run.** Without `-for` there is no `trend`; sample a minute. +- **Treating `notable` as a verdict.** It is a deterministic shortlist; check each item against the + question. Garbage-collected heaps (`*heap*`, `jvm_memory_used`) never get trend notes: they + climb until each GC. Look for leaks in `process_resident_memory_bytes`. +- **Calling a load swing a fault.** When traffic, CPU, latency and pools all rise or fall by a + similar amount together, the load changed; a short window may catch one phase of a cycle. + Faults show up as `down`, `errors`, `near_limit`, or one series moving against the rest. +- **Reading `errors.share` from a `-state` file recorded with `-l`.** The file keeps only the + series `record` selected, so the share is of those (`1` for `-l 'code=~5..'`). A live run + takes the share from the whole metric whatever `-l` says. +- **Missing a metric.** It may be hidden: check `totals.hidden`, rerun with `-a`. +- **Missing series.** `series_omitted > 0` means more exist: narrow with `-l` or use `-top 0`. +- **Empty `notable` = healthy?** Only for what the heuristics cover; still read the metrics the + user asked about. diff --git a/skills/analyzing-metrics-with-promtop/reference.md b/skills/analyzing-metrics-with-promtop/reference.md new file mode 100644 index 0000000..b2e0edb --- /dev/null +++ b/skills/analyzing-metrics-with-promtop/reference.md @@ -0,0 +1,86 @@ +# `promtop.summary/v1` reference + +Written by `promtop render -format summary` as one line of JSON. Fields are left out when they +have no value (no NaN, no null). The `schema` string changes only on incompatible changes. + +```json +{ + "schema": "promtop.summary/v1", + "source": "http://localhost:8080/metrics", + "target": "shop/orders 1.4.2", + "window": {"from": "2026-10-29T14:40:49Z", "to": "2026-10-29T14:41:49Z", "samples": 31, "interval_s": 2}, + "totals": {"metrics": 30, "series": 58, "shown": 26, "hidden": 4}, + "notable": [ + {"metric": "demo_dependency_up", "labels": {"dependency": "kafka"}, "reason": "down", "changes": 2}, + {"metric": "demo_disk_avail_bytes", "labels": {"mount": "/var/lib/demo"}, "value": "used_ratio", "reason": "near_limit", "last": 1}, + {"metric": "demo_http_requests_total", "value": "rate", "reason": "errors", "last": 0.5, "share": 0.0076}, + {"metric": "demo_queue_depth", "labels": {"queue": "emails"}, "value": "value", "reason": "rising", + "last": 18, "from": 11.5, "to": 19.5, "change": 0.70} + ], + "notable_omitted": 0, + "metrics": [ + {"name": "demo_http_requests_total", "type": "counter", "kind": "rate", "unit": "1/s", "value": "rate", + "help": "Total HTTP requests.", "series_count": 15, + "sum": {"last": 126, "avg": 120, "min": 96.4, "max": 155, "trend": "flat", "change": 0.04}, + "series": [{"labels": {"code": "200", "method": "GET", "path": "/api/orders"}, + "last": 43.9, "avg": 41.2, "min": 30.1, "max": 47.9, "trend": "flat", "change": 0.02, "total": 342186}], + "series_omitted": 14} + ] +} +``` + +## Top level + +| Field | Meaning | +|---|---| +| `source`, `target` | the scraped URL or file; service name and version from `target_info`, if any | +| `window` | first and latest scrape (RFC 3339, UTC), number of samples kept, seconds between samples | +| `totals.metrics`, `totals.series` | what was read: the whole target, or what `-m`/`-f`/`-l` selected (live series) | +| `totals.shown` | metrics in this document; `totals.hidden` were left out, `-a` includes them | +| `notable`, `notable_omitted` | at most 20 notes, most urgent first; how many more there were. Absent with `-list` | +| `metrics` | absent with `-notable` | + +## Notes + +| `reason` | When | Extra fields | +|---|---|---| +| `down` | a 0/1 health flag (a name with `up`, `success`, `ready`, `healthy`, `alive`, `connected`…) is 0 | `changes`: flips in the window | +| `near_limit` | usage at 90% or more | `last`: used ratio | +| `errors` | a counter with a 5xx `code`/`status`/`status_code` label is moving | `last`: 5xx rate summed over the metric, `share` of the metric's whole rate, before `-l` (from a `-state` file: of the series `record` kept). No `labels`: the series are in `metrics` | +| `expires_soon` | a timestamp metric named like a deadline (`*expir*`, `not_after`, `notafter`, `deadline`) is less than 7 days ahead | `in_s` | +| `flapped` | a health flag is up now but changed during the window, or a stateset switched (one note per metric, `labels` = the state now) | `changes` | +| `changed` | a timestamp metric moved: restart (`process_start_time_seconds`), deploy, reload | | +| `rising`, `falling` | mean of the window's last third differs from the first by 50% or more, or starts from 0 | `last`, `from`, `to`, `change` | + +A series' `trend` needs 3 samples; a trend *note* is stricter. Event counters must move beyond +Poisson noise (3σ of the counts in a third). Gauges, quantiles and other rates must move more than +they wobble within the thirds (Welch, 3σ), which needs 6 samples (`-for 12s` at the default +`-i 2s`); quantiles also need 20 observations per third. Garbage-collected heap gauges (names with +`heap`, `go_memstats_alloc_bytes`, `go_memory_used*`, `jvm_memory_used*`) get no trend notes: they +climb until each collection. + +## Metrics + +| Field | Meaning | +|---|---| +| `type` | exposition type: `counter`, `gauge`, `histogram`, `summary`, `info`, `stateset`, `untyped` | +| `kind` | how promtop reads it: `rate`, `gauge`, `histogram`, `summary`, `usage`, `state`, `time`, `info` | +| `unit` | base unit of the series' numbers: `seconds`, `bytes`, `ratio`, `percent`, a symbol (`°C`, `V`), `1/s`, `bytes/s`; absent for plain numbers | +| `value` | what `last/avg/min/max` describe: `rate`, `value`, `used_ratio`, or a quantile (`p99`) | +| `labels` | labels shared by every series, left out of the series | +| `limit` | usage: the metric holding the maximum (`node_filesystem_size_bytes`) | +| `series_count`, `series_omitted` | live series; how many `-top` cut (series are sorted: problems and biggest values first) | +| `sum` | rates with several series: their total over the window | + +## Series + +| Field | Kinds | Meaning | +|---|---|---| +| `labels` | all | own labels | +| `last`, `avg`, `min`, `max` | rate, gauge, histogram, summary, usage | of the metric's `value` over the window | +| `trend`, `change` | same, 3+ samples | `rising`/`falling`/`flat`; relative change, first third → last third | +| `total` | rate | the counter's lifetime value | +| `rate`, `mean`, `quantiles` | histogram, summary | observations per second, average observation, latest quantiles | +| `used`, `limit` | usage | latest, in `unit` | +| `up` / `active` | state | health flag is 1 / stateset member is set; `changes` = flips in the window | +| `time`, `age_s` / `in_s` | time | the timestamp; seconds since, or seconds until when it is ahead |