Skip to content

log_warnings spike — suspected bad deploy: 306ac09 #539

Closed
Closed
log_warnings spike — suspected bad deploy: 306ac09#539
Labels
anomalyBaseline-relative runtime anomalybot:vahterFindings for the vahter botmonitorRuntime anomaly finding from the monitor agentpriority-high

Description

@github-actions
github-actions
bot
opened on Sep 30, 2026

Problem

log_warnings_24h is significantly elevated in the trailing-24h window compared to its baseline.

Evidence

log_warnings_24h: current=68 median_7d=17 median_28d=16 ratio_vs_28d=4.25 z_score_28d=8.231747923303264 history_days_28d=28 low_confidence=false

Baseline comparison

current=68 median_7d=17 median_28d=16 ratio_vs_28d=4.25 z_score_28d=8.231747923303264 history_days_28d=28

Change correlation

Suspected bad deploy: commit 306ac09 / "fix(vahter): keep autoban classification lookups correlated (#530)" touched src/VahterBanBot/ within the preceding 72h. ArgoCD recorded two deploys in the window: revision 41949a4c48ae4254d81e3254fec8878b5e4db998 deployed at 2026-09-28T07:17:12Z and revision 415ce2dac0ba4167ff600632400d032e25dfcd57 deployed at 2026-09-28T12:01:29Z.

Activity

  1. github-actions
    added
    priority-high
    bot:vahterFindings for the vahter bot
    monitorRuntime anomaly finding from the monitor agent
    anomalyBaseline-relative runtime anomaly
    on Sep 30, 2026

  2. github-actions commented on Sep 30, 2026

    @github-actions
    github-actionsBot
    on Sep 30, 2026 – with GitHub Actions
    Author

    Problem

    Elevated warning volume for log_warnings_24h remains active in this run.

    Evidence

    log_warnings_24h series (from the evidence bundle):
    current=69, median_7d=17, median_28d=17, ratio_vs_28d=4.0588235294117645, z_score_28d=7.039753776383079, history_days_28d=29, low_confidence=false, max_28d=68

    Baseline comparison

    current=69 median_7d=17 median_28d=17 ratio_vs_28d=4.0588235294117645 z_score_28d=7.039753776383079 history_days_28d=29

    Change correlation

    Correlated — suspected bad deploy: merge 306ac09 / "fix(vahter): keep autoban classification lookups correlated" (PR #530) touched src/VahterBanBot/ in the preceding 72h.
    ArgoCD deploys in preceding 72h:

    • id=206 revision=41949a4c48ae4254d81e3254fec8878b5e4db998 deployedAt=2026-09-28T07:17:12Z
    • id=207 revision=415ce2dac0ba4167ff600632400d032e25dfcd57 deployedAt=2026-09-28T12:01:29Z

    This is an error-series anomaly (dormant_exempt=false). Per monitor rules, correlated deploys must be filed — updating the existing open fingerprint with fresh evidence.

    Run timestamp: 2026-09-30T04:38:18Z

  3. github-actions commented on Sep 30, 2026

    @github-actions
    github-actionsBot
    on Sep 30, 2026 – with GitHub Actions
    Author

    Problem

    Elevated warnings count: log_warnings_24h outside baseline.

    Evidence

    log_warnings_24h: current=71 median_7d=17 median_28d=17 ratio_vs_28d=4.176470588235294 z_score_28d=6.387785636790477 history_days_28d=29

    Change correlation

    Suspected bad deploy: ArgoCD revision 9f5a637e (deploy id=208, deployedAt=2026-09-30T07:24:33Z).
    Merges to main touching src/VahterBanBot in the preceding 72h:

  4. github-actions commented on Sep 30, 2026

    @github-actions
    github-actionsBot
    on Sep 30, 2026 – with GitHub Actions
    Author

    Automated monitor agent comment (run timestamp: 2026-09-30T16:36:49Z)

    Problem

    Elevated warnings count: log_warnings_24h outside baseline.

    Evidence

    log_warnings_24h: current=65, median_7d=17, median_28d=16.5, ratio_vs_28d=3.9393939393939394, z_score_28d=4.6578796539016665, history_days_28d=29

    Baseline comparison

    current=65 median_7d=17 median_28d=16.5 ratio_vs_28d=3.9393939393939394 z_score_28d=4.6578796539016665 history_days_28d=29

    Change correlation

    Correlated — suspected bad deploy: merge 5e7ec91 (PR #538, "Add daily moderation quality statistics with late-event repair") touched src/VahterBanBot/ within the preceding 72h. ArgoCD deploy id=209 (revision fdc5421e77240d278cee8cdc79b030151d34ef49) deployedAt 2026-09-30T10:55:55Z is within 72h and may have rolled this change.

  5. github-actions commented on Sep 30, 2026

    @github-actions
    github-actionsBot
    on Sep 30, 2026 – with GitHub Actions
    Author

    Problem

    Recurring elevated log_warnings_24h outside baseline — currently active.

    Evidence

    log_warnings_24h: current=50 median_7d=17 median_28d=16.5 ratio_vs_28d=3.0303030303030303 z_score_28d=2.932891594074934 history_days_28d=28 low_confidence=false

    Baseline comparison

    current=50 median_7d=17 median_28d=16.5 ratio_vs_28d=3.0303030303030303 z_score_28d=2.932891594074934 history_days_28d=28

    Change correlation

    Correlated — suspected bad deploy: merges to src/VahterBanBot/ in the preceding 72h:

    ArgoCD deploys to vahter-bot in the preceding 72h:

    • id=210 revision=85d2ef9e... deployedAt=2026-09-30T17:42:21Z
    • id=209 revision=fdc5421e... deployedAt=2026-09-30T10:55:55Z
    • id=208 revision=9f5a637e... deployedAt=2026-09-30T07:24:33Z

    Because this is an error-series finding (warnings/errors), a nearby merge/deploy raises severity — file as correlated — suspected bad deploy.

  6. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsBot
    on Oct 1, 2026 – with GitHub Actions
    Author

    Problem

    First transition back within baseline for previously-filed error-burst log_warnings_24h (issue #539).

    Evidence

    log_warnings_24h: current=14, median_7d=17, median_28d=16.5, ratio_vs_28d=0.8484848484848485, z_score_28d=-0.4977323657451609, history_days_28d=28, low_confidence=false

    Baseline comparison

    current=14 median_7d=17 median_28d=16.5 ratio_vs_28d=0.8484848484848485 z_score_28d=-0.4977323657451609 history_days_28d=28

    Change correlation

    Merges touching src/VahterBanBot/ in the preceding 72h:

    ArgoCD deploys in the preceding 72h: id=206..210 (revisions 41949a4c..., 415ce2d..., 9f5a637..., fdc5421e..., 85d2ef9e...) — present.

    I checked change_context. There were merges and ArgoCD deploys in the preceding 72h that could plausibly explain past increases; however today's evidence shows log_warnings_24h is below its 28-day median. As this is an existing error-series fingerprint, I am posting a clean status comment rather than filing a new issue.

  7. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsBot
    on Oct 1, 2026 – with GitHub Actions
    Author

    Monitor update — vahter — 2026-10-01T08:42:53Z

    Sources

    prometheus: ok
    loki: ok
    argocd: ok

    Evidence (verbatim from bundle)

    log_warnings_24h: current=14 median_7d=17 median_28d=16 ratio_vs_28d=0.875 z_score_28d=-0.48799294167915797 history_days_28d=29 low_confidence=false

    Assessment

    This run reads clean for the log_warnings_24h signal. The known open fingerprint vahter/monitor/error-burst/log_warnings_24h shows consecutive_clean_days=1 and days_since_last_active=0 (mechanical). Posting the first transition-back comment marking the signal as clean.

  8. github-actions commented on Oct 2, 2026

    @github-actions
    github-actionsBot
    on Oct 2, 2026 – with GitHub Actions
    Author

    Monitor heartbeat — vahter — 2026-10-02T01:08:51Z

    Sources

    prometheus: ok
    loki: ok
    argocd: ok

    Evidence (verbatim from bundle)

    log_warnings_24h: current=22 median_7d=17 median_28d=16 ratio_vs_28d=1.375 z_score_28d=0.25414307885489534 history_days_28d=28 low_confidence=false

    Assessment

    This run reads clean for the log_warnings_24h signal (below anomaly thresholds). Continuing to wait out the 7-day clean bar before closing the fingerprint.

  9. github-actions commented on Oct 3, 2026

    @github-actions
    github-actionsBot
    on Oct 3, 2026 – with GitHub Actions
    Author

    Monitor agent heartbeat — signal clean

    Evidence
    log_warnings_24h: current=29, median_7d=21, median_28d=17, ratio_vs_28d=1.7058823529411764, z_score_28d=0.8638056771395488, history_days_28d=28, low_confidence=false

    Mechanical (from fingerprint): last_active_at=2026-09-30T20:35:21Z, days_since_last_active=2, consecutive_clean_days=2

    Sources: prometheus/loki/argocd: ok

    Posting daily heartbeat while waiting to reach 7 consecutive clean days before closing.

  10. github-actions commented on Oct 4, 2026

    @github-actions
    github-actionsBot
    on Oct 4, 2026 – with GitHub Actions
    Author

    Monitor heartbeat — vahter — 2026-10-04T10:27:32Z

    Sources

    prometheus: ok
    loki: ok
    argocd: ok

    Evidence

    log_warnings_24h: current=18 median_7d=24 median_28d=17 ratio_vs_28d=1.0588235294117647 z_score_28d=-0.20421805398296033 history_days_28d=29 low_confidence=false

    Assessment

    This run reads clean for the log_warnings_24h signal (within baseline). The known open fingerprint vahter/monitor/error-burst/log_warnings_24h reports days_since_last_active=3 and consecutive_clean_days=3. Posting daily heartbeat while waiting to reach 7 consecutive clean days before closing.

  11. github-actions commented on Oct 5, 2026

    @github-actions
    github-actionsBot
    on Oct 5, 2026 – with GitHub Actions
    Author

    Monitor heartbeat — vahter — run timestamp: 2026-10-05T08:52:37Z

    Sources

    prometheus: ok loki: ok argocd: ok

    Evidence (verbatim)

    log_warnings_24h: current=24 median_7d=24 median_28d=18.5 ratio_vs_28d=1.2972972972972974 z_score_28d=0.3169772246444546 history_days_28d=29 low_confidence=false

    Assessment

    Signal reads clean this run. Mechanical: last_active_at=2026-09-30T20:35:21Z days_since_last_active=4 consecutive_clean_days=4.
    Posting daily heartbeat while waiting to reach 7 consecutive clean days before closing.

  12. github-actions commented on Oct 5, 2026

    @github-actions
    github-actionsBot
    on Oct 5, 2026 – with GitHub Actions
    Author

    Monitor heartbeat — vahter — run 2026-10-05T20:45:22Z

    Evidence (verbatim)

    log_warnings_24h: current=9, median_7d=24, median_28d=19, ratio_vs_28d=0.47368421052631576, z_score_28d=-1.061520443985427, history_days_28d=28, low_confidence=false

    Baseline comparison

    current=9 median_7d=24 median_28d=19 ratio_vs_28d=0.47368421052631576 z_score_28d=-1.061520443985427 history_days_28d=28

    Change correlation

    • merges to main touching src/VahterBanBot/ in preceding 72h: none
    • ArgoCD deploy history (preceding 72h): none

    Mechanical status

  13. github-actions commented on Oct 6, 2026

    @github-actions
    github-actionsBot
    on Oct 6, 2026 – with GitHub Actions
    Author

    Monitor heartbeat — vahter — run 2026-10-06T01:07:46Z

    Evidence (verbatim)

    log_warnings_24h: current=10, median_7d=23.5, median_28d=18.5, ratio_vs_28d=0.5405405405405406, z_score_28d=-0.9605658772451788, history_days_28d=28, low_confidence=false

    Baseline comparison

    current=10 median_7d=23.5 median_28d=18.5 ratio_vs_28d=0.5405405405405406 z_score_28d=-0.9605658772451788 history_days_28d=28

    Change correlation

    • merges to main touching src/VahterBanBot/ in preceding 72h: none
    • ArgoCD deploy history (preceding 72h): none

    Mechanical status

  14. github-actions commented on Oct 7, 2026

    @github-actions
    github-actionsBot
    on Oct 7, 2026 – with GitHub Actions
    Author

    Signal 'log_warnings_24h' is within baseline this run.

    Evidence

    log_warnings_24h: current=17, median_7d=23, median_28d=17.5, ratio_vs_28d=0.9714285714285714, z_score_28d=-0.2713764133974802, history_days_28d=28, low_confidence=false

    Mechanical state from fingerprints: days_since_last_active=6, consecutive_clean_days=6 — still below the 7-day close bar; will close after 7 consecutive clean days.

    Run timestamp

    2026-10-07T01:12:14Z

  15. github-actions commented on Oct 7, 2026

    @github-actions
    github-actionsBot
    on Oct 7, 2026 – with GitHub Actions
    Author

    Closing: vahter/monitor/error-burst/log_warnings_24h

    The issue's mechanical fields show days_since_last_active: 7 and consecutive_clean_days: 7. Today's evidence confirms the series is within baseline.

    Baseline comparison:
    current=22 median_7d=22 median_28d=17 ratio_vs_28d=1.2941176470588236 z_score_28d=0.21705652371713877 history_days_28d=28

    Known issue metadata:

    Action: Closing this issue because mechanical 7-day bar is met and today's live evidence confirms the signal is clean.

  16. github-actions
    closed this as completedon Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    anomalyBaseline-relative runtime anomalybot:vahterFindings for the vahter botmonitorRuntime anomaly finding from the monitor agentpriority-high

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions