Grafana sync: attribute logs by node address, not validator_name (prerequisite for uptime-points automation)

[rasca]

Problem

GrafanaValidatorStatusService (backend/validators/grafana_service.py) determines ValidatorWallet.logs_status by bridging logs through the metrics validator_name label, which silently mis-reports operators that ship logs but not metrics.

Current flow in sync_network():

1. Prometheus query keyed by validator_name, node:
   max by(validator_name, node) (genlayer_node_info{job="prometheus.scrape.genlayer_node", network="<label>"})
2. Loki query keyed by the free-text label:
   sum by (validator_name) (count_over_time({job="genlayer-node"} | json | network="<label>" [5m]))
3. Join: addr -> validator_name from the metrics frame, then
   logs_ok = bool(vname and log_counts.get(vname, 0) > 0).

Two structural bugs follow:

• Logs-without-metrics are invisible. If a node ships logs but not metrics, there is no metrics series to map its address to a validator_name, so vname is None and logs_ok is False even though logs are flowing. Verified live (2026-07-01): BlockPro (node 0x7745607927aa9ba77e6b131dc7986aca2ae43235, Asimov) ships ~1,500 log lines / 5 min to Loki but no metrics — the sync marks it logs_status='shame', which is wrong. At audit time 7 operators were in this state: BlockPro, Coinage-x-DAIC, HusoNode, Stakely, UnityNodes, VJNNODES, plus one node with the default name.
• validator_name is not a key. It is operator-chosen free text. Operators leave the docs defaults (MyValidator / VALIDATOR_NAME=MyValidator, NODE_ID=validator-001) or use names that differ from their on-chain moniker ([NODERS] vs Noders, AURORAX-asimov vs AURORAX, P-OPS Team vs pops-team). Any join through it can mis-attribute one operator's logs to another.

The reliable join key exists: every log line's JSON body contains node (validator wallet address) and operator (operator wallet address). Example real log line fields: {"node":"0x7745607927AA9BA77E6b131dC7986aCA2aE43235","operator":"0xbeE1f2962C0B0ED58Ef0B682CD0cC90bCD5393eE","network":"asimov-phase5",...}. The Foundation Grafana dashboard (gl-wall-of-shame, monitoring repo) was already migrated to this join and it works.

Why now

We are about to start awarding/docking uptime points from metrics_status/logs_status (announced to operators). With the current join, BlockPro-type operators would be wrongly penalized for "missing logs" they are actually shipping. This fix is a hard prerequisite for that automation.

Fix

All in backend/validators/grafana_service.py:

1. Loki query — group by node in _build_query_body():

   sum by (node) (count_over_time({job="genlayer-node"} | json | network="<label>" [5m]))
   

   Optional hardening: append | __error__="" after | json to drop lines that fail JSON parsing (a handful exist in production; label filters after | json already exclude them implicitly, but the explicit guard is cheaper and clearer).

2. parse_response() — parse the Loki frames' node label (instead of validator_name) into a set/dict keyed by lowercased address. Keep returning the prom side keyed by lowercased node as today. The validator_name_by_address map can stay if anything still displays it, but it must no longer participate in the logs decision.

3. sync_network() — decide logs directly:

   addr_lower = (wallet.address or '').lower()
   metrics_ok = addr_lower in prom_addresses
   logs_ok = log_counts_by_addr.get(addr_lower, 0) > 0   # no vname bridge

4. Case-handling: metric/log label values are checksummed addresses; ValidatorWallet.address is stored lowercase. Lowercase the label values at parse time (this is the same convention as the Grafana dashboard: "lowercase BOTH sides before joining").

Tests

backend/validators/tests/test_grafana_service.py (GrafanaParseResponseTests, GrafanaSyncNetworkTests) currently encode the validator_name bridge — update the fixtures to Loki frames labeled by node, and add the regression case this bug is about:

• BlockPro-shaped fixture: a wallet whose address appears in the Loki frames but NOT in the Prometheus frames → expect metrics_status='shame', logs_status='on'.
• Inverse fixture (metrics only, no logs) → metrics_status='on', logs_status='shame'.
• Case fixture: checksummed node label vs lowercase wallet address still matches.

Run: cd backend && source env/bin/activate && python manage.py test validators.tests.test_grafana_service

Coordination / gotchas

• Branch off dev (PRs target dev, not main). The in-flight branch feat/grafana-shame-history-streaks also touches grafana_service.py (Grafana-observed version sync, ValidatorWalletObservation/snapshot writes) — rebase/coordinate if it has merged; the logs-join change is orthogonal to the version work.
• GRAFANA_NETWORK_LABELS maps asimov -> asimov-phase5, bradbury -> bradbury-phase1; no change needed there.
• The sync runs via cron-protected POST /api/v1/validators/wallets/sync-grafana/ (X-Cron-Token) on a GitHub Actions cron that effectively fires every 1–3 h — after deploy, wait for one run (or trigger with the token) before judging results.
• Repo conventions: 3-layer commit messages per .claude/commands/commit.md, no attribution lines, add a one-line CHANGELOG entry under ## Unreleased.

Acceptance criteria

• After one sync, GET /api/v1/validators/wallets/wall-of-shame/ shows BlockPro (Asimov) with logs_status='on' and metrics_status='shame' (assuming its live state is unchanged).
• No wallet's logs decision depends on validator_name anywhere in grafana_service.py.
• Existing on/shame transitions (*_shame_started_at bookkeeping) behave as before for unaffected wallets.
• Tests above pass.