diff --git a/AGENTS.md b/AGENTS.md index 9d35cf4d..b9513bea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -105,6 +105,7 @@ When editing or adding skills in this repo, follow these rules (and add new skil ## CI / validation gotchas - The test suite expects **every directory under `skills/`** to be listed in a marketplace. If you add a new skill (or rebase onto a main branch that added skills), update the appropriate marketplace file or CI will fail with `Skills missing from marketplace: [...]`. +- `skills/index.js` embeds the **full text** of every `SKILL.md`. Any commit that modifies a `skills/*/SKILL.md` must also include a regenerated `skills/index.js` (run `node scripts/build-skills-catalog.mjs`), or `test_index_is_up_to_date` will fail. - `scripts/sync_extensions.py` keeps generated artifacts in sync: Claude Code command files, README catalog section, coverage checks, and vendor symlinks. Run `python scripts/sync_extensions.py --check` (or just push — CI runs it) to verify everything is consistent. Run without `--check` to auto-fix. The "Quick Start" section in `README.md` (OpenHands SDK, Claude Code, and Codex setup instructions) is **manually maintained** above the auto-generated catalog markers and is intentionally not generated by the sync script. - The sync script uses PyYAML to parse SKILL.md frontmatter. If you add a skill with a slash trigger (e.g., `triggers: ["/mycommand"]`), the script auto-generates `commands/mycommand.md`. **Note:** Slash triggers in SKILL.md frontmatter are deprecated — prefer adding a `commands/command-name.md` file to the plugin's `commands/` directory instead. Keyword triggers (non-slash) remain the recommended way to activate skills by topic. diff --git a/README.md b/README.md index 8c198bc3..5dbc74c0 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ See [`integrations/README.md`](integrations/README.md), [`automations/README.md` ## Extensions Catalog -This repository contains **2 marketplace(s)** with **57 extensions** (47 skills, 10 plugins). +This repository contains **2 marketplace(s)** with **58 extensions** (48 skills, 10 plugins). ### large-codebase @@ -79,7 +79,7 @@ OpenHands skills for interacting, improving, and refactoring large codebases Official skills and plugins for OpenHands — the open-source AI software engineer. -**53 extensions** (45 skills, 8 plugins) +**54 extensions** (46 skills, 8 plugins) | Name | Type | Description | Commands | |------|------|-------------|----------| @@ -93,6 +93,7 @@ Official skills and plugins for OpenHands — the open-source AI software engine | code-review | skill | Rigorous code review focusing on data structures, simplicity, security, pragmatism, and risk/safety evaluation. Provi... | `/codereview`, `/codereview-roasted` | | code-simplifier | skill | Simplifies and refines code across three dimensions - code reuse, code quality, and efficiency - while preserving all... | `/simplify` | | datadog | skill | Query and analyze Datadog logs, metrics, APM traces, and monitors using the Datadog API. Use when debugging productio... | — | +| datadog-error-monitor | skill | Create a cron automation that polls Datadog logs every 15 minutes, maintains a self-evolving regex-based error patter... | `/datadog-monitor:setup` | | deno | skill | Common project operations using Deno (tasks, run/test/lint/fmt, and dependency management). | — | | discord | skill | Build and automate Discord integrations (bots, webhooks, slash commands, and REST API workflows). Use when the user m... | — | | docker | skill | Run Docker commands within a container environment, including starting the Docker daemon and managing containers. Use... | — | diff --git a/automations/catalog/datadog-error-monitor.json b/automations/catalog/datadog-error-monitor.json new file mode 100644 index 00000000..6b632aaa --- /dev/null +++ b/automations/catalog/datadog-error-monitor.json @@ -0,0 +1,11 @@ +{ + "id": "datadog-error-monitor", + "name": "Datadog error monitor", + "category": "Developer tools", + "description": "Poll Datadog logs for new or spiking errors, investigate root causes in your codebase, and post a Slack summary with optional PR fixes.", + "requiredIntegrationIds": ["datadog", "slack"], + "popularityRank": 80, + "estimatedSetupMinutes": 10, + "prompt": "/datadog-monitor:setup", + "exampleImplementation": "Trigger: cron, every 15 minutes\nRequired secrets: DD_API_KEY, DD_APP_KEY, SLACK_BOT_TOKEN\n\n1. Query Datadog logs using a configured filter (e.g. service:api status:error).\n2. Match log events against a regex-based error pattern library stored in a state file.\n3. Track hit counts per pattern; detect spikes (count > 3× rolling baseline).\n4. TODO: configure Datadog query, Slack alert channel, and local repo paths.\n5. When new or spiking errors are found, start an investigation conversation.\n6. The agent categorizes unknown errors into named patterns, investigates root causes in the configured codebases, creates a PR if confident, and posts a Slack summary." +} diff --git a/automations/index.js b/automations/index.js index 4c9ab120..5b3cea88 100644 --- a/automations/index.js +++ b/automations/index.js @@ -5,6 +5,7 @@ import slack_channel_monitor from "./catalog/slack-channel-monitor.json" with { import linear_triage_assistant from "./catalog/linear-triage-assistant.json" with { type: "json" }; import research_brief_writer from "./catalog/research-brief-writer.json" with { type: "json" }; import incident_retrospective_drafter from "./catalog/incident-retrospective-drafter.json" with { type: "json" }; +import datadog_error_monitor from "./catalog/datadog-error-monitor.json" with { type: "json" }; export const AUTOMATION_CATALOG = [ github_pr_reviewer, @@ -14,5 +15,6 @@ export const AUTOMATION_CATALOG = [ linear_triage_assistant, research_brief_writer, incident_retrospective_drafter, + datadog_error_monitor, ]; export default AUTOMATION_CATALOG; diff --git a/integrations/catalog/datadog.json b/integrations/catalog/datadog.json new file mode 100644 index 00000000..955d85d8 --- /dev/null +++ b/integrations/catalog/datadog.json @@ -0,0 +1,27 @@ +{ + "id": "datadog", + "name": "Datadog", + "description": "Query logs, metrics, APM traces, and monitors via the Datadog REST API.", + "docsUrl": "https://docs.datadoghq.com/api/latest/", + "iconBg": "#632CA6", + "keywords": [ + "monitoring", + "logs", + "metrics", + "observability", + "apm", + "errors", + "alerts" + ], + "kind": "http", + "defaultConnectionOptionId": "api", + "connectionOptions": [ + { + "id": "api", + "provider": "http", + "auth": { + "strategy": "api_key" + } + } + ] +} diff --git a/marketplaces/openhands-extensions.json b/marketplaces/openhands-extensions.json index 75218f46..82b8d7e3 100644 --- a/marketplaces/openhands-extensions.json +++ b/marketplaces/openhands-extensions.json @@ -5,7 +5,7 @@ "email": "contact@all-hands.dev" }, "metadata": { - "description": "Official skills and plugins for OpenHands \u2014 the open-source AI software engineer.", + "description": "Official skills and plugins for OpenHands — the open-source AI software engineer.", "maintainer": "OpenHands", "homepage": "https://github.com/OpenHands/extensions" }, @@ -13,7 +13,7 @@ { "name": "agent-creator", "source": "./skills/agent-creator", - "description": "Create file-based sub-agents as Markdown files \u2014 no Python code required. Guides the user through a structured interview and generates a ready-to-deploy .md agent file following the OpenHands SDK specification.", + "description": "Create file-based sub-agents as Markdown files — no Python code required. Guides the user through a structured interview and generates a ready-to-deploy .md agent file following the OpenHands SDK specification.", "category": "development", "keywords": [ "agent", @@ -165,6 +165,23 @@ "apm" ] }, + { + "name": "datadog-error-monitor", + "source": "./skills/datadog-error-monitor", + "description": "Create a cron automation that polls Datadog logs every 15 minutes, maintains a self-evolving regex-based error pattern library, and triggers an OpenHands investigation conversation when new or spiking errors are detected. The agent categorizes unknown errors, investigates root causes in local codebases, optionally creates PRs, and posts a summary to Slack.", + "category": "monitoring", + "keywords": [ + "datadog", + "monitoring", + "errors", + "logs", + "alerting", + "cron", + "automation", + "slack", + "investigation" + ] + }, { "name": "deno", "source": "./skills/deno", @@ -391,7 +408,7 @@ { "name": "openhands", "source": "./plugins/openhands", - "description": "Unified OpenHands plugin \u2014 bundles Cloud CLI, REST API (openhands-api), and Automations (openhands-automation) into a single plugin.", + "description": "Unified OpenHands plugin — bundles Cloud CLI, REST API (openhands-api), and Automations (openhands-automation) into a single plugin.", "category": "openhands", "keywords": [ "openhands", @@ -432,7 +449,7 @@ { "name": "pr-review", "source": "./plugins/pr-review", - "description": "Automated PR code review \u2014 analyzes diffs and posts inline review comments via the GitHub API.", + "description": "Automated PR code review — analyzes diffs and posts inline review comments via the GitHub API.", "category": "code-quality", "keywords": [ "pr-review", @@ -444,7 +461,7 @@ { "name": "qa-changes", "source": "./plugins/qa-changes", - "description": "Validate pull request changes by actually running the code \u2014 setting up the environment, exercising changed behavior, and posting a structured QA report.", + "description": "Validate pull request changes by actually running the code — setting up the environment, exercising changed behavior, and posting a structured QA report.", "category": "quality-assurance", "keywords": [ "qa", @@ -593,7 +610,7 @@ { "name": "iterate", "source": "./skills/iterate", - "description": "Iterate on a GitHub pull request \u2014 drive it through CI, code review, and QA until merge-ready. Monitors state, fixes failures, addresses review feedback, retries flaky checks, and pushes fixes in one continuous loop.", + "description": "Iterate on a GitHub pull request — drive it through CI, code review, and QA until merge-ready. Monitors state, fixes failures, addresses review feedback, retries flaky checks, and pushes fixes in one continuous loop.", "category": "productivity", "keywords": [ "github", diff --git a/skills/datadog-error-monitor/.claude-plugin b/skills/datadog-error-monitor/.claude-plugin new file mode 120000 index 00000000..665797f0 --- /dev/null +++ b/skills/datadog-error-monitor/.claude-plugin @@ -0,0 +1 @@ +.plugin \ No newline at end of file diff --git a/skills/datadog-error-monitor/.codex-plugin b/skills/datadog-error-monitor/.codex-plugin new file mode 120000 index 00000000..665797f0 --- /dev/null +++ b/skills/datadog-error-monitor/.codex-plugin @@ -0,0 +1 @@ +.plugin \ No newline at end of file diff --git a/skills/datadog-error-monitor/.plugin/plugin.json b/skills/datadog-error-monitor/.plugin/plugin.json new file mode 100644 index 00000000..721b737c --- /dev/null +++ b/skills/datadog-error-monitor/.plugin/plugin.json @@ -0,0 +1,23 @@ +{ + "name": "datadog-error-monitor", + "version": "1.0.0", + "description": "Create a cron automation that polls Datadog logs every 15 minutes, maintains a self-evolving regex-based error pattern library, and triggers an OpenHands investigation conversation when new or spiking errors are detected. The agent categorizes unknown errors, investigates root causes in local codebases, optionally creates PRs, and posts a summary to Slack.", + "author": { + "name": "OpenHands", + "email": "contact@all-hands.dev" + }, + "homepage": "https://github.com/OpenHands/extensions", + "repository": "https://github.com/OpenHands/extensions", + "license": "MIT", + "keywords": [ + "datadog", + "monitoring", + "errors", + "logs", + "alerting", + "cron", + "automation", + "slack", + "investigation" + ] +} diff --git a/skills/datadog-error-monitor/README.md b/skills/datadog-error-monitor/README.md new file mode 100644 index 00000000..51c0243e --- /dev/null +++ b/skills/datadog-error-monitor/README.md @@ -0,0 +1,92 @@ +# datadog-error-monitor + +An OpenHands skill that creates an automated Datadog error monitoring loop. + +## What it does + +A cron automation runs every 15 minutes, querying Datadog logs with a +pre-configured filter. It maintains a library of known error patterns (regex-based) +and their historical hit counts. When something new or anomalous is detected, it +starts an OpenHands investigation conversation that diagnoses the problem, +optionally creates a PR fix, and posts a summary to Slack. + +``` +Every 15 min + │ + ▼ +Query Datadog ──► Match against known patterns ──► Update run_history + │ + ├── Unknown logs? ──────────────────────────────────────────────┐ + │ │ + └── Any pattern spiked (count > 3× baseline)?──────────────────┤ + │ + Start one + investigation + conversation + │ + ┌────────────────┘ + │ + Agent categorizes errors, + investigates code, creates PR + if confident, posts to Slack +``` + +## Prerequisites + +- OpenHands running in local mode +- Datadog credentials: `DD_API_KEY` + `DD_APP_KEY` + `DD_SITE` +- Slack bot token: `SLACK_BOT_TOKEN` with `chat:write` scope +- Local code repositories (already cloned) for root-cause investigation +- Git host token (`GITHUB_PERSONAL_ACCESS_TOKEN`, `GITLAB_TOKEN`, or `BITBUCKET_TOKEN`) for PR creation + +## Setup + +Load this skill in OpenHands and it will guide you through a setup conversation: + +1. Verify Datadog credentials and confirm your site region +2. Define the log query (tested live against your Datadog org) +3. Configure the Slack channel for summaries +4. Register local repository paths and git host tokens +5. Tune optional parameters (max log samples, examples per pattern, spike threshold) +6. Dry-run the query and confirm before deploying + +The skill then packages and deploys the automation automatically. + +## How patterns work + +The pattern library is empty on first run — the system builds it organically: + +- **First run:** All logs are uncategorized. The investigation agent groups them + into named patterns with regex matchers and writes these to the state file. +- **Subsequent runs:** Logs are matched against known patterns. Only spikes or + genuinely new error types trigger an investigation. +- **Over time:** The pattern library grows and stabilises. Investigations become + rarer and more targeted. + +Patterns are stored in the state file at: +``` +~/.openhands/workspaces/automation-state/dd_monitor_{automation_id}.json +``` + +You can inspect and edit this file at any time to rename patterns, fix regexes, +or remove stale entries. Deleting the file resets the system. + +## Token efficiency + +The script itself is deterministic — no LLM calls on quiet runs. An OpenHands +conversation is only started when one of these conditions is met: +- At least one log event matched no known pattern +- At least one pattern's count exceeded `mean(last 3 runs) × spike_multiplier` + +One conversation handles all triggers from a given run. A second conversation +cannot start until the first finishes. + +## Files + +| File | Purpose | +|---|---| +| `SKILL.md` | Step-by-step setup workflow for the OpenHands agent | +| `scripts/main.py` | Cron script template (customised at automation-creation time) | +| `references/state-schema.md` | State file JSON schema and agent write protocol | +| `references/datadog-api.md` | Datadog API reference (auth, log search, rate limits) | +| `references/agent-prompt-template.md` | Investigation prompt structure and token budget notes | diff --git a/skills/datadog-error-monitor/SKILL.md b/skills/datadog-error-monitor/SKILL.md new file mode 100644 index 00000000..2f598335 --- /dev/null +++ b/skills/datadog-error-monitor/SKILL.md @@ -0,0 +1,374 @@ +--- +name: datadog-error-monitor +description: > + This skill should be used when the user asks to "monitor Datadog for errors", + "watch Datadog logs for error spikes", "investigate Datadog errors automatically", + "alert on new or recurring errors in Datadog", or "set up automated error + monitoring with Datadog and Slack". Guides the user through creating a cron + automation that polls Datadog logs every 15 minutes, maintains a library of + known error patterns, and triggers an OpenHands investigation conversation when + new or spiking errors are detected. The investigation agent categorizes unknown + errors, investigates root causes in local codebases, optionally creates PRs, + and posts a concise summary to Slack. +triggers: + - /datadog-monitor:setup +--- + +# Datadog Error Monitor + +Create a cron automation that polls Datadog logs every 15 minutes. + +On each run the script: +1. Queries Datadog using a configured log filter (e.g. `service:(api OR worker) status:error`) +2. Matches log events against a library of known error patterns (regex-based, stored in a state file) +3. Tracks hit counts per pattern across runs +4. When **new/uncategorized logs** or a **significant count spike** is detected, + starts a single OpenHands investigation conversation + +When an investigation conversation runs it: +- Categorizes uncategorized logs into named patterns and writes them to the state file +- Investigates root causes in the configured local codebases +- Creates a PR only when highly confident of a code-level fix +- Posts a concise summary to the configured Slack channel + +> **Local mode only.** Monitored code repositories must be cloned on the same +> machine running OpenHands. + +--- + +## Prerequisites + +### Required secrets + +Verify all of the following are set in **OpenHands Settings → Secrets**: + +| Secret name | Description | +|---|---| +| `DD_API_KEY` | Datadog API key | +| `DD_APP_KEY` | Datadog Application key (required for log search — distinct from the API key) | +| `SLACK_BOT_TOKEN` | Slack bot token with `chat:write` scope | + +Check with: +```bash +[ -n "$DD_API_KEY" ] && echo "DD_API_KEY is set" || echo "DD_API_KEY is NOT set" +[ -n "$DD_APP_KEY" ] && echo "DD_APP_KEY is set" || echo "DD_APP_KEY is NOT set" +[ -n "$SLACK_BOT_TOKEN" ] && echo "SLACK_BOT_TOKEN is set" || echo "SLACK_BOT_TOKEN is NOT set" +``` + +If any are missing, inform the user and stop — the automation cannot function without them. + +Git host token(s) are confirmed per-repository in Step 5. + +### Optional secret + +| Secret name | Default | Purpose | +|---|---|---| +| `OPENHANDS_URL` | `http://localhost:8000` | Base URL for conversation links | + +--- + +## Setup Workflow + +Follow these steps in order. + +### Step 1 — Verify Datadog credentials + +```bash +DD_SITE_TMP="datadoghq.com" +curl -s -X POST "https://api.${DD_SITE_TMP}/api/v2/logs/events/search" \ + -H "DD-API-KEY: $DD_API_KEY" \ + -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ + -H "Content-Type: application/json" \ + -d '{"filter":{"query":"*","from":"now-1m","to":"now"},"page":{"limit":1}}' \ + | python3 -c " +import json, sys +d = json.load(sys.stdin) +if 'errors' in d: + print('ERROR:', d['errors']) +elif 'data' in d: + print('OK — credentials valid, got', len(d['data']), 'sample event(s)') +else: + print('Unexpected response:', list(d.keys())) +" +``` + +- If `DD_API_KEY` or `DD_APP_KEY` is missing → tell the user and stop. +- If the API returns a 403 → tell the user to check both keys in Datadog: + *Organization Settings → API Keys* (for `DD_API_KEY`) and + *Organization Settings → Application Keys* (for `DD_APP_KEY`). + +### Step 2 — Confirm Datadog site + +Ask: *"What is your Datadog site?* +*(Press Enter for the default: `datadoghq.com`)* +*Other options: `datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`"* + +Record as `DD_SITE`. Default: `"datadoghq.com"`. + +Re-run the Step 1 credential check substituting the confirmed site. + +### Step 3 — Collect the Datadog log query + +Ask: *"What Datadog log query should be monitored? This filter runs every 15 minutes.* +*Example: `service:(deploy OR runtime-api) status:error`* +*Example: `service:payment-api (@http.status_code:5* OR status:error)`* +*Tip: develop and test your query in the Datadog Logs Explorer first."* + +Run a test query to show the user what they will be monitoring: +```bash +USER_QUERY="REPLACE_WITH_USER_QUERY" +DD_SITE="REPLACE_WITH_DD_SITE" +curl -s -X POST "https://api.${DD_SITE}/api/v2/logs/events/search" \ + -H "DD-API-KEY: $DD_API_KEY" \ + -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ + -H "Content-Type: application/json" \ + -d "{ + \"filter\": {\"query\": \"${USER_QUERY}\", \"from\": \"now-15m\", \"to\": \"now\"}, + \"sort\": \"-timestamp\", + \"page\": {\"limit\": 5} + }" | python3 -c " +import json, sys +d = json.load(sys.stdin) +if 'errors' in d: + print('ERROR:', d['errors']) + sys.exit(1) +events = d.get('data', []) +print(f'{len(events)} event(s) in the last 15 minutes (showing up to 5):') +for e in events: + attrs = e.get('attributes', {}) + ts = attrs.get('timestamp', '') + msg = (attrs.get('message') or attrs.get('error', {}).get('message') or '')[:120] + svc = attrs.get('service', '') + print(f' [{ts}] [{svc}] {msg}') +" +``` + +Show the output to the user. If the query looks correct, record as `DD_QUERY`. +If they want to adjust it, iterate until they are satisfied. + +### Step 4 — Collect Slack configuration + +Ask: *"Which Slack channel should investigation summaries be posted to?* +*Provide the channel name (e.g. `#errors-alerts`) or ID (e.g. `C0123456789`)."* + +Verify the bot token: +```bash +curl -s https://slack.com/api/auth.test \ + -H "Authorization: Bearer $SLACK_BOT_TOKEN" \ + | python3 -c "import json,sys; d=json.load(sys.stdin); print('Bot:', d.get('user')) if d.get('ok') else print('ERROR:', d.get('error'))" +``` + +If the user provided a channel name, resolve it to an ID: +```bash +CHANNEL_NAME="REPLACE_WITH_NAME_WITHOUT_HASH" +curl -s "https://slack.com/api/conversations.list?types=public_channel,private_channel&limit=200&exclude_archived=true" \ + -H "Authorization: Bearer $SLACK_BOT_TOKEN" \ + | python3 -c " +import json, sys +data = json.load(sys.stdin) +if not data.get('ok'): + print('ERROR:', data.get('error')) + sys.exit(1) +for ch in data.get('channels', []): + if ch['name'] == '${CHANNEL_NAME}'.lstrip('#'): + print(ch['id']) + break +else: + print('Channel not found — provide the channel ID directly (right-click channel → Copy link)') +" +``` + +Record as `SLACK_CHANNEL_ID`. + +### Step 5 — Collect monitored repositories + +Ask: *"Which local code repositories should the investigation agent have access to?* +*These must already be cloned on this machine.* +*For each, provide the absolute path and the git host (GitHub / GitLab / Bitbucket).* +*Example: `/home/user/projects/my-api` (GitHub)"* + +For each repository: + +**a) Verify the path is a git repo:** +```bash +cd /path/to/repo && git remote -v && git log --oneline -1 +``` + +**b) Confirm the git host token is available:** + +| Host | Required secret | Verification | +|---|---|---| +| GitHub | `GITHUB_PERSONAL_ACCESS_TOKEN` | `curl -s https://api.github.com/user -H "Authorization: Bearer $GITHUB_PERSONAL_ACCESS_TOKEN" \| python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('login', d.get('message')))"` | +| GitLab | `GITLAB_TOKEN` | `curl -s https://gitlab.com/api/v4/user -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \| python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('username', d.get('message')))"` | +| Bitbucket | `BITBUCKET_TOKEN` | `curl -s https://api.bitbucket.org/2.0/user --user "$BITBUCKET_TOKEN" \| python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('username', d.get('error')))"` | + +If the required token is missing, inform the user. The investigation agent +can still investigate code but will not be able to create PRs until the token +is added to OpenHands Settings → Secrets. + +**c) Extract the remote identifier** from `git remote get-url origin`. + +Build `REPO_CONFIGS` as a Python list literal: +```python +REPO_CONFIGS: list[dict] = [ + {"path": "/path/to/repo1", "host": "github", "remote": "owner/repo1"}, + {"path": "/path/to/repo2", "host": "gitlab", "remote": "owner/repo2"}, + {"path": "/path/to/repo3", "host": "bitbucket", "remote": "owner/repo3"}, +] +``` + +### Step 6 — Tune parameters + +Ask about the following settings. Offer the defaults and accept Enter to keep them: + +| Parameter | Default | Prompt | +|---|---|---| +| `MAX_UNKNOWN_LOGS` | `100` | *"Max uncategorized log samples to send to the agent per run?"* | +| `EXAMPLES_PER_PATTERN` | `3` | *"How many example logs to keep per error pattern?"* | +| `SPIKE_MULTIPLIER` | `3.0` | *"Spike threshold — alert when count exceeds this multiple of the rolling 3-run average?"* | + +### Step 7 — Dry run confirmation + +Run the Step 3 query one final time with a 15-minute window and show the user: +- Total number of matching events +- The first 3 example log messages + +Ask: *"This is what the automation will monitor every 15 minutes. Does it look right?* +*Shall I create the automation?"* + +If the user wants changes, return to Step 3. + +### Step 8 — Generate the automation script + +Read `scripts/main.py` from this skill's directory and **copy it verbatim**. +Apply exactly the following constant substitutions near the top of the file: + +> **Do not reimplement or hand-write a replacement script.** Only substitute the +> configuration constants below — all other logic must remain unchanged. + +| Placeholder | Replace with | +|---|---| +| `DD_QUERY = "service:(deploy OR runtime-api) status:error"` | `DD_QUERY = "{dd_query}"` | +| `DD_SITE = "datadoghq.com"` | `DD_SITE = "{dd_site}"` | +| `SLACK_CHANNEL_ID = "C0123456789"` | `SLACK_CHANNEL_ID = "{channel_id}"` | +| `REPO_CONFIGS: list[dict] = []` | `REPO_CONFIGS: list[dict] = {repo_configs_literal}` | +| `MAX_UNKNOWN_LOGS = 100` | `MAX_UNKNOWN_LOGS = {max_unknown}` | +| `EXAMPLES_PER_PATTERN = 3` | `EXAMPLES_PER_PATTERN = {examples_per}` | +| `SPIKE_MULTIPLIER = 3.0` | `SPIKE_MULTIPLIER = {multiplier}` | +| `DEFAULT_OPENHANDS_URL = "http://localhost:8000"` | `DEFAULT_OPENHANDS_URL = "{url}"` (keep default if user has no preference) | + +Write the script to a temporary build directory: +```bash +mkdir -p /tmp/dd-monitor-build +# copy and substitute scripts/main.py → /tmp/dd-monitor-build/main.py +``` + +Validate syntax and confirm substitutions: +```bash +python3 -m py_compile /tmp/dd-monitor-build/main.py && echo "Syntax OK" +grep -n 'DD_QUERY = ' /tmp/dd-monitor-build/main.py +grep -n 'DD_SITE = ' /tmp/dd-monitor-build/main.py +grep -n 'SLACK_CHANNEL_ID = ' /tmp/dd-monitor-build/main.py +grep -n 'REPO_CONFIGS' /tmp/dd-monitor-build/main.py +grep -n 'def get_secret' /tmp/dd-monitor-build/main.py +grep -n 'def main' /tmp/dd-monitor-build/main.py +``` + +If any check fails, re-copy the template and redo the substitutions. + +### Step 9 — Package and upload + +Determine the Automation backend URL from the `` block in +your system context. + +```bash +tar -czf /tmp/dd-monitor.tar.gz -C /tmp/dd-monitor-build . + +OPENHANDS_HOST="" + +TARBALL_PATH=$(curl -s -X POST \ + "${OPENHANDS_HOST}/api/automation/v1/uploads?name=datadog-error-monitor" \ + -H "X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY" \ + -H "Content-Type: application/gzip" \ + --data-binary @/tmp/dd-monitor.tar.gz \ + | python3 -c "import json,sys; print(json.load(sys.stdin)['tarball_path'])") + +echo "Uploaded: $TARBALL_PATH" +``` + +### Step 10 — Create the automation + +```bash +curl -s -X POST "${OPENHANDS_HOST}/api/automation/v1" \ + -H "X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY" \ + -H "Content-Type: application/json" \ + -d "{ + \"name\": \"Datadog Error Monitor\", + \"trigger\": {\"type\": \"cron\", \"schedule\": \"*/15 * * * *\"}, + \"tarball_path\": \"$TARBALL_PATH\", + \"entrypoint\": \"python3 main.py\", + \"timeout\": 120 + }" | python3 -m json.tool +``` + +Record the returned `id`. + +### Step 11 — Confirm and hand off + +Tell the user: + +> ✅ **Datadog Error Monitor** is running! +> +> - **Automation ID:** `{id}` +> - **Query:** `{dd_query}` +> - **Site:** `{dd_site}` +> - **Alert channel:** `{slack_channel}` +> - **Repos:** `{repo_paths}` +> - **Schedule:** every 15 minutes (`*/15 * * * *`) +> - **State file:** `~/.openhands/workspaces/automation-state/dd_monitor_{id}.json` +> +> **What happens next:** +> The first run will treat all matching logs as uncategorized and start an +> investigation conversation. The agent will build the initial pattern library. +> After a few runs, the system converges on a stable baseline and only alerts +> on new error types or significant spikes. +> +> You can inspect or edit the state file at any time to adjust patterns, remove +> stale ones, or reset the system entirely by deleting the file. + +--- + +## Runtime Behaviour (per 15-min poll) + +Each cron run executes `main.py` with a 120-second timeout: + +1. **Loads state** — picks up any pattern updates written by the previous agent run +2. **Queries Datadog** — window: `[last_poll − 60 s overlap, now − 10 s]` +3. **Matches logs** — each event tested against `known_patterns` regexes +4. **Updates history** — appends counts to each pattern's `run_history` (capped at 20); refreshes `examples` and `last_seen` +5. **Checks active conversation** — if one is running, saves updated state and exits; spike evaluation is skipped until the investigation finishes +6. **Evaluates triggers:** + - Unknown logs (count ≥ 1) → trigger + - Pattern spike (`current > mean(last 3 runs) × SPIKE_MULTIPLIER`) → trigger +7. **If triggered:** builds the investigation prompt and starts one OpenHands conversation; records its ID in state +8. **Saves state** and fires the completion callback + +The investigation agent (asynchronous): +- Categorizes unknown logs into named patterns → writes back to state file +- Investigates spiking patterns in the configured repos +- Creates PRs only if highly confident of a code-level fix +- Posts a Slack summary with findings + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| `Datadog API error 403` | `DD_APP_KEY` invalid or wrong site | Verify both keys in Datadog Org Settings; re-check `DD_SITE` | +| No events returned | Query too narrow, wrong site, or no recent errors | Test query in Datadog Logs Explorer UI | +| Agent creates very specific patterns | LLM being over-literal | Edit state file to merge narrow patterns; the regex quality rules in the prompt help prevent this | +| Spike keeps re-triggering | Pattern regex too broad (every log matches) | Tighten the regex in the state file; valid Python regex only | +| Investigation never finishes | Agent stuck on a complex codebase | View conversation in OpenHands UI; clear `active_conversation` in state file to unblock next run | +| Slack not receiving messages | `SLACK_BOT_TOKEN` missing `chat:write` or bot not in channel | Re-install app with correct scope; invite bot to channel | +| State file write conflict | Script and agent writing concurrently (rare) | At worst one count update is lost; recovers automatically next run | diff --git a/skills/datadog-error-monitor/commands/datadog-monitor-setup.md b/skills/datadog-error-monitor/commands/datadog-monitor-setup.md new file mode 100644 index 00000000..9e360f8a --- /dev/null +++ b/skills/datadog-error-monitor/commands/datadog-monitor-setup.md @@ -0,0 +1,8 @@ +--- +# auto-generated by sync_extensions.py +description: This skill should be used when the user asks to "monitor Datadog for errors", "watch Datadog logs for error spikes", "investigate Datadog errors automatically", "alert on new or recurring errors in Datadog", or "set up automated error monitoring with Datadog and Slack". Guides the user through creating a cron automation that polls Datadog logs every 15 minutes, maintains a library of known error patterns, and triggers an OpenHands investigation conversation when new or spiking errors are detected. The investigation agent categorizes unknown errors, investigates root causes in local codebases, optionally creates PRs, and posts a concise summary to Slack. +--- + +Read and follow the complete instructions in the SKILL.md file located in this skill's directory. + +$ARGUMENTS diff --git a/skills/datadog-error-monitor/references/agent-prompt-template.md b/skills/datadog-error-monitor/references/agent-prompt-template.md new file mode 100644 index 00000000..72e4221b --- /dev/null +++ b/skills/datadog-error-monitor/references/agent-prompt-template.md @@ -0,0 +1,157 @@ +# Agent Investigation Prompt + +This document describes the structure of the prompt that `main.py` sends to the +OpenHands investigation conversation. The prompt is generated dynamically on each +trigger run. Understanding it helps with debugging, tuning, and extending the skill. + +--- + +## When the Prompt is Sent + +A new conversation is created (and this prompt sent) when — and only when — **all** +of the following are true: + +1. No investigation conversation is currently active (`active_conversation` is `null`) +2. At least one trigger condition is met: + - **Unknown logs:** one or more log events did not match any known pattern + - **Spike:** at least one known pattern's current count exceeds + `mean(last 3 runs) × SPIKE_MULTIPLIER` + +A single conversation handles all triggers from that run together. + +--- + +## Prompt Structure + +The prompt is a Markdown document with four tasks: + +### Task 1 — Categorize unknown logs + +Shown only if `total_unknown > 0`. Contains: +- The total count of uncategorized logs detected +- A notice if the total exceeds `MAX_UNKNOWN_LOGS` (only a sample is shown) +- The sampled log messages, numbered, wrapped in `` tags +- The path to the state file where new patterns should be written +- Detailed instructions on the required pattern format + +**Pattern writing format** the agent must follow: + +```json +{ + "known_patterns": { + "": { + "name": "...", + "regex": "...", + "run_history": [], + "last_seen": "", + "examples": [{"timestamp": "...", "message": "..."}] + } + } +} +``` + +The agent reads the current state file, merges in the new patterns, and writes it +back. It must preserve all existing patterns and the `last_poll_timestamp` and +`active_conversation` fields. + +**Pattern quality guidance** embedded in the prompt: +- Avoid encoding timestamps, request IDs, memory addresses, PIDs, or UUIDs in regexes +- Prefer fewer, broader patterns over many narrow ones +- A good pattern matches the error *class*, not a single specific occurrence +- Use `.*` to skip variable parts between fixed anchors + +### Task 2 — Investigate spiking patterns + +Shown only if at least one pattern has a count spike. For each spiking pattern: +- Pattern name and regex +- Current count vs. recent baseline (formatted as `mean(last N runs)`) +- Up to `EXAMPLES_PER_PATTERN` example log messages (truncated to 300 chars each) + +### Task 3 — Update and fix + +Contains: +- List of repository paths with git host and remote identifier +- Instruction to `git pull` on each repo before investigating +- Decision gate: + - **Code-level bugs:** create a PR only if highly confident + - **Infrastructure/config/data issues:** describe findings, no PR + +### Task 4 — Post Slack summary + +Contains: +- The `SLACK_CHANNEL_ID` embedded in the script +- Required summary content: new categories, spike diagnoses, PRs or suggestions +- The `chat.postMessage` curl template +- Instruction to skip posting if no findings (no unknown logs, no spikes) + +--- + +## Token Budget Considerations + +The prompt size is bounded by these configurable limits: + +| Parameter | Default | Controls | +|---|---|---| +| `MAX_UNKNOWN_LOGS` | 100 | Max uncategorized log messages included in the prompt | +| `EXAMPLES_PER_PATTERN` | 3 | Max example logs shown per spiking pattern | +| `MAX_LOG_MESSAGE_CHARS` | 500 | Max chars per log message in the prompt | + +At defaults with 100 unknown logs and 5 spiking patterns with 3 examples each, +the prompt is roughly 50–70 KB — comfortably within context limits for modern LLMs. + +If a query is returning a very high volume of uncategorized logs every run, +reduce `MAX_UNKNOWN_LOGS` during setup, or tighten the Datadog query filter +to reduce noise. + +--- + +## Investigation Conversation Workspace + +The agent conversation is started with `working_dir` set to the first configured +repository path (or a dedicated `dd-monitor-investigations/` directory if no +repos are configured). All user secrets and MCP configuration are forwarded so +the agent has the same tool access as any other OpenHands conversation. + +The agent uses `terminal` and `file_editor` tools to: +- Navigate repository code (`cd`, `grep`, `git log`) +- Pull latest changes (`git pull`) +- Read and write the state file (to add new patterns) +- Create branches and PRs using the appropriate git host CLI + +--- + +## Example Prompt Excerpt + +```markdown +# Datadog Error Monitor — Investigation Request + +## Context +- **Datadog query:** `service:(api OR worker) status:error` +- **Time window:** 2024-03-15T14:00:00Z → 2024-03-15T14:14:50Z +- **State file path:** `/home/user/.openhands/workspaces/automation-state/dd_monitor_abc123.json` + +## Your Tasks + +### Task 1 — Categorize unknown logs +There are **47 uncategorized log events** in this window. + +For each distinct error type you identify, add a new entry to `known_patterns` +in the state file: `/home/user/.openhands/workspaces/automation-state/dd_monitor_abc123.json` + +... + + +[1] Connection refused to postgres://db.internal:5432/production at pool.js:88 +[2] ETIMEDOUT: request to https://payments.stripe.com/v1/charges timed out after 30000ms +... +[47] Cannot read property 'id' of undefined at UserController.update (user.js:214) + + +### Task 2 — Investigate spiking patterns + +#### Pattern: `Redis connection timeout in CacheService` +- Current count: **47** +- Recent baseline: 1.3 (last 3 runs: [2, 1, 1]) +- Regex: `(?:redis|Redis).*(?:timeout|connection refused).*CacheService` +... +``` diff --git a/skills/datadog-error-monitor/references/datadog-api.md b/skills/datadog-error-monitor/references/datadog-api.md new file mode 100644 index 00000000..342354d3 --- /dev/null +++ b/skills/datadog-error-monitor/references/datadog-api.md @@ -0,0 +1,166 @@ +# Datadog API Reference + +## Authentication + +Every request requires both headers: + +``` +DD-API-KEY: {DD_API_KEY} +DD-APPLICATION-KEY: {DD_APP_KEY} +``` + +Both secrets are fetched from the OpenHands secrets store at runtime. The +Application Key (`DD_APP_KEY`) is required for log search — the API key alone +is not sufficient. + +### Datadog sites + +The API base URL depends on the Datadog region configured during setup: + +| Site | Base URL | +|---|---| +| US1 (default) | `https://api.datadoghq.com` | +| EU1 | `https://api.datadoghq.eu` | +| US3 | `https://api.us3.datadoghq.com` | +| US5 | `https://api.us5.datadoghq.com` | +| AP1 | `https://api.ap1.datadoghq.com` | + +`DD_SITE` is embedded in `main.py` at automation-creation time. + +--- + +## Log Search (v2) + +### Endpoint + +``` +POST https://api.{DD_SITE}/api/v2/logs/events/search +``` + +### Request body + +```json +{ + "filter": { + "query": "service:(api OR worker) status:error", + "from": "2024-03-15T14:00:00Z", + "to": "2024-03-15T14:15:00Z" + }, + "sort": "timestamp", + "page": { "limit": 1000 } +} +``` + +| Field | Description | +|---|---| +| `filter.query` | Datadog log search query (same syntax as the Logs UI) | +| `filter.from` | ISO 8601 UTC start (inclusive) | +| `filter.to` | ISO 8601 UTC end (exclusive) | +| `sort` | `"timestamp"` (oldest first) or `"-timestamp"` (newest first) | +| `page.limit` | Results per page, max 1000 | + +**Time shortcuts** (`now-15m`, `now-1h`, etc.) are valid but the monitor uses +absolute ISO 8601 timestamps to ensure deterministic overlap windows. + +### Response structure + +```json +{ + "data": [ + { + "id": "AAAAAWgN8Xwgr1vKDQAAAABBWWdOOFh3RzFmQ0FBQUFBQUFBQUFBQUFBQUFBQWc", + "type": "log", + "attributes": { + "timestamp": "2024-03-15T14:07:33Z", + "status": "error", + "service": "api", + "message": "Unhandled exception in PaymentService", + "error": { + "message": "NullPointerException at PaymentService.java:127", + "stack": "java.lang.NullPointerException\n at com.example.PaymentService.processPayment(PaymentService.java:127)\n ..." + }, + "host": "prod-api-01", + "tags": ["env:production", "version:2.3.1"] + } + } + ], + "meta": { + "page": { "after": "cursor-for-next-page" } + } +} +``` + +### Useful query syntax + +| Pattern | Example | +|---|---| +| Service filter | `service:(api OR worker OR payment-service)` | +| Status filter | `status:error` | +| Multiple conditions | `service:api status:error @http.status_code:5*` | +| Text search | `"connection refused"` | +| Tag filter | `env:production` | +| Exclude a service | `service:* -service:health-check` | +| Error with stack | `status:error @error.stack:*NullPointer*` | + +Test queries in the [Datadog Logs Explorer](https://app.datadoghq.com/logs) before configuring them here. + +### Rate limits + +- Log search: **300 requests/hour** per organization +- At one poll per 15 minutes = 4 requests/hour — well within limits +- Each response is limited to 1,000 events; the monitor uses this maximum + +--- + +## Log Message Extraction + +The monitor extracts a representative message string from each log event using +this priority order: + +1. `attributes.message` +2. `attributes.error.message` +3. `attributes.msg` + +If a stack trace is present at `attributes.error.stack`, the first 200 chars +are appended to the message. The combined result is truncated to 500 chars. + +--- + +## Credential Verification + +Use this to verify credentials before setting up the automation: + +```bash +DD_SITE="datadoghq.com" +curl -s -X POST "https://api.${DD_SITE}/api/v2/logs/events/search" \ + -H "DD-API-KEY: $DD_API_KEY" \ + -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ + -H "Content-Type: application/json" \ + -d '{"filter":{"query":"*","from":"now-1m","to":"now"},"page":{"limit":1}}' \ + | python3 -c " +import json, sys +d = json.load(sys.stdin) +if 'errors' in d: + print('ERROR:', d['errors']) +elif 'data' in d: + print('OK — credentials valid') +else: + print('Unexpected response:', list(d.keys())) +" +``` + +### Common errors + +| HTTP status | Meaning | Fix | +|---|---|---| +| `403 Forbidden` | `DD_APP_KEY` missing or invalid | Check Application Key in Datadog → Org Settings → Application Keys | +| `400 Bad Request` | Malformed query or date format | Validate query in Datadog Logs UI | +| `429 Too Many Requests` | Rate limit exceeded | Reduce poll frequency or check for other automation hitting the same key | + +--- + +## Further Reading + +- [Logs Search API v2](https://docs.datadoghq.com/api/latest/logs/#search-logs) +- [Log Search Syntax](https://docs.datadoghq.com/logs/explorer/search_syntax/) +- [API & Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/) diff --git a/skills/datadog-error-monitor/references/state-schema.md b/skills/datadog-error-monitor/references/state-schema.md new file mode 100644 index 00000000..2a1fb8db --- /dev/null +++ b/skills/datadog-error-monitor/references/state-schema.md @@ -0,0 +1,158 @@ +# State File Schema + +The monitor persists all state between cron runs in a single JSON file. + +## Location + +``` +~/.openhands/workspaces/automation-state/dd_monitor_{automation_id}.json +``` + +The path is derived at runtime from the `WORKSPACE_BASE` environment variable +(two levels up, into `automation-state/`). The file is created on the first run +and is safe to delete — the monitor will rebuild from scratch, treating all logs +as uncategorized on the next run. + +--- + +## Top-level Fields + +| Field | Type | Description | +|---|---|---| +| `version` | `int` | Schema version (currently `1`) | +| `last_poll_timestamp` | `string` (ISO 8601 UTC) | Upper bound of the last successful Datadog query. The next query starts here (minus a 60 s overlap). | +| `active_conversation` | `object \| null` | The single in-flight OpenHands investigation conversation, or `null` if none is running. | +| `known_patterns` | `object` | Map of pattern UUID → pattern definition. Updated by the investigation agent between runs. | + +--- + +## `active_conversation` Object + +Present when an investigation is in progress; `null` otherwise. The script +checks conversation status on every run and clears this field when the +conversation reaches a terminal state (`idle`, `finished`, `error`, `stuck`). + +While `active_conversation` is non-null, the script skips trigger evaluation — +logs are still fetched and pattern history is still updated, but no new +conversation is started. + +| Field | Type | Description | +|---|---|---| +| `id` | `string` | OpenHands conversation ID | +| `started_at` | `string` (ISO 8601 UTC) | When the conversation was created | +| `trigger_summary` | `string` | Human-readable description of what triggered the investigation | +| `status` | `string` | Last known status (`"running"`) — updated by the script when it checks | + +--- + +## `known_patterns` Object + +Keys are UUIDs assigned when a pattern is first created (by the investigation +agent). Values are pattern definition objects. + +**The investigation agent is responsible for adding new entries** to this object +when it categorizes uncategorized logs. The script reads the file at the start +of every run and picks up any patterns written by the agent. + +### Pattern Definition + +| Field | Type | Description | +|---|---|---| +| `name` | `string` | Concise human-readable label, e.g. `"Redis connection timeout in AuthService"` | +| `regex` | `string` | Python regex (`re.IGNORECASE \| re.DOTALL`) matched against the extracted log message. Must be broad enough to catch future occurrences — avoid encoding timestamps, request IDs, memory addresses, or UUIDs. | +| `run_history` | `list[int]` | Hit counts from the most recent runs (oldest first). Capped at 20 entries. Updated by the script every run, even when the count is 0. | +| `last_seen` | `string \| null` (ISO 8601 UTC) | Timestamp of the most recent matching log. `null` if never seen since pattern was added. | +| `examples` | `list[object]` | Most recent matching log snippets (up to `EXAMPLES_PER_PATTERN`, configurable). Oldest entries are dropped as new ones arrive. | + +### Example Object (inside `examples`) + +| Field | Type | Description | +|---|---|---| +| `timestamp` | `string` | Log event timestamp from Datadog | +| `message` | `string` | Truncated log message (max 500 chars), including a stack trace prefix if present | + +--- + +## Agent Protocol for Writing New Patterns + +When the investigation agent identifies a new error category, it **reads** the +current state file, adds the new pattern under `known_patterns`, and **writes** +the file back atomically. The agent must: + +1. Generate a UUID for the pattern key (Python: `import uuid; str(uuid.uuid4())`) +2. Set `run_history` to a list containing the count from this investigation window +3. Set `last_seen` to the current UTC timestamp +4. Add up to `EXAMPLES_PER_PATTERN` examples + +The agent must **not** overwrite any existing `known_patterns` entries, and must +preserve all other top-level fields (`last_poll_timestamp`, `active_conversation`). + +### Concurrent Write Safety + +The script runs every 15 minutes with a ~120 s timeout. The investigation agent +runs asynchronously and typically takes longer. Since the script completes well +before the agent finishes, write conflicts are unlikely. In the rare case of +overlap, the last writer wins on the state file — at worst, one run's pattern +count update is lost. This recovers automatically on the next poll. + +--- + +## Annotated Example + +```json +{ + "version": 1, + "last_poll_timestamp": "2024-03-15T14:30:00Z", + "active_conversation": null, + "known_patterns": { + "a1b2c3d4-e5f6-7890-abcd-ef1234567890": { + "name": "Redis connection timeout in CacheService", + "regex": "(?:redis|Redis).*(?:timeout|connection refused|ECONNREFUSED).*CacheService", + "run_history": [0, 2, 0, 0, 1, 47, 39], + "last_seen": "2024-03-15T14:28:41Z", + "examples": [ + { + "timestamp": "2024-03-15T14:28:41Z", + "message": "Error: Redis connection refused at CacheService.get (cache.js:42)\n at RedisClient.connect (redis.js:91)" + }, + { + "timestamp": "2024-03-15T14:15:03Z", + "message": "Redis timeout after 5000ms at CacheService.set (cache.js:58)" + } + ] + }, + "f0e1d2c3-b4a5-6789-0abc-def123456789": { + "name": "Unhandled NullPointerException in PaymentService", + "regex": "NullPointerException.*at.*PaymentService\\.(process|charge|refund)", + "run_history": [3, 3, 4, 2, 3, 3, 4], + "last_seen": "2024-03-15T14:27:19Z", + "examples": [ + { + "timestamp": "2024-03-15T14:27:19Z", + "message": "java.lang.NullPointerException\n at com.example.PaymentService.processPayment(PaymentService.java:127)" + } + ] + } + } +} +``` + +--- + +## Lifecycle Notes + +- **First run:** `known_patterns` is empty. All logs are uncategorized. An + investigation conversation is started unconditionally (assuming at least one + log matches the query). The agent builds the initial pattern library. + +- **Steady state:** The script matches logs, increments run_history, and only + triggers if an unknown log appears or a pattern count spikes above the + rolling baseline × `SPIKE_MULTIPLIER`. + +- **Retired patterns:** Patterns are never automatically removed. A pattern with + consistently zero counts for many runs is harmless — it just accumulates zeros + in `run_history`. If a fixed bug no longer produces logs, its count drops to + zero and stays there. You can manually remove patterns from the state file. + +- **Resetting:** Delete the state file to reset everything. The next run will + treat the system as fresh — all logs uncategorized, no baseline history. diff --git a/skills/datadog-error-monitor/scripts/main.py b/skills/datadog-error-monitor/scripts/main.py new file mode 100644 index 00000000..033ecd5b --- /dev/null +++ b/skills/datadog-error-monitor/scripts/main.py @@ -0,0 +1,820 @@ +""" +Datadog Error Monitor — OpenHands Automation Script + +Polls Datadog for log errors on a cron schedule. On each run: + 1. Queries Datadog logs using a pre-configured filter query. + 2. Matches each log against known error patterns (regex). + 3. Tracks hit counts per pattern across runs. + 4. If unknown/uncategorized logs are detected OR a pattern count spikes + significantly, creates a single OpenHands investigation conversation. + 5. The conversation agent categorizes unknown logs, investigates root causes, + optionally creates PRs, and posts a summary to Slack. + +Configuration constants are embedded at automation-creation time by the skill. +See SKILL.md for the full setup workflow. + +Required secrets (set in OpenHands Settings → Secrets): + DD_API_KEY — Datadog API key + DD_APP_KEY — Datadog Application key (required for log search) + SLACK_BOT_TOKEN — Slack bot token (chat:write scope) + +Optional secret: + OPENHANDS_URL — base URL for conversation links (default: http://localhost:8000) +""" + +import json +import os +import re +import sys +from pathlib import Path +import urllib.error +import urllib.request +from datetime import datetime, timedelta, timezone + +# ── Embedded configuration (filled in by the skill at creation time) ────────── +DD_QUERY = "service:(deploy OR runtime-api) status:error" +DD_SITE = "datadoghq.com" +SLACK_CHANNEL_ID = "C0123456789" +REPO_CONFIGS: list[dict] = [] # [{"path": "/path/to/repo", "host": "github", "remote": "owner/repo"}] +MAX_UNKNOWN_LOGS = 100 +EXAMPLES_PER_PATTERN = 3 +SPIKE_MULTIPLIER = 3.0 +DEFAULT_OPENHANDS_URL = "http://localhost:8000" + +# ── Internal constants ───────────────────────────────────────────────────────── +INITIAL_LOOKBACK_MINUTES = 15 # lookback window on the very first run +OVERLAP_SECONDS = 60 # extend each query back by this to avoid boundary gaps +MIN_RUNS_FOR_SPIKE = 3 # minimum run_history entries before spike detection activates +MAX_LOG_MESSAGE_CHARS = 500 # truncate individual extracted log messages at this length +MAX_RUN_HISTORY = 20 # keep at most this many entries in run_history per pattern +INVESTIGATION_BUDGET = 10 # total tool calls the agent may spend across all investigation tasks +PATTERN_ARCHIVE_DAYS = 30 # patterns not seen within this many days are moved to the archive +STUCK_CONVERSATION_MINUTES = 45 # conversations still 'running' beyond this are treated as stuck + + +# ── Stdlib-only helpers ──────────────────────────────────────────────────────── + +def _env_api_key() -> str: + return ( + os.environ.get("SESSION_API_KEY") + or os.environ.get("OH_SESSION_API_KEYS_0") + or "" + ) + + +def get_secret(name: str) -> str: + """Fetch a named secret stored in the agent server.""" + url = os.environ.get("AGENT_SERVER_URL", "").rstrip("/") + key = _env_api_key() + req = urllib.request.Request( + f"{url}/api/settings/secrets/{name}", + headers={"X-Session-API-Key": key}, + ) + with urllib.request.urlopen(req, timeout=15) as r: + return r.read().decode().strip() + + +def fire_callback( + status: str = "COMPLETED", + error: str | None = None, + conversation_id: str | None = None, +) -> None: + """Signal run completion to the automation service. MUST be called on every exit path.""" + url = os.environ.get("AUTOMATION_CALLBACK_URL", "") + if not url: + return + body: dict = {"status": status, "run_id": os.environ.get("AUTOMATION_RUN_ID", "")} + if error: + body["error"] = error + if conversation_id: + body["conversation_id"] = conversation_id + req = urllib.request.Request( + url, + data=json.dumps(body).encode(), + headers={ + "Content-Type": "application/json", + "Authorization": f"Bearer {os.environ.get('AUTOMATION_CALLBACK_API_KEY', '')}", + }, + ) + try: + urllib.request.urlopen(req, timeout=10) + except Exception as exc: + print(f"Callback error (non-fatal): {exc}") + + +# ── State management ─────────────────────────────────────────────────────────── + +def _state_file_path() -> str: + """Derive a persistent storage path stable across cron runs. + + WORKSPACE_BASE = {root}/automation-runs/{run_id} + State lives at {root}/automation-state/dd_monitor_{automation_id}.json + """ + workspace_base = os.environ.get("WORKSPACE_BASE", "") + event_payload = json.loads(os.environ.get("AUTOMATION_EVENT_PAYLOAD", "{}")) + automation_id = event_payload.get("automation_id", "default") + + if workspace_base: + root = Path(workspace_base).resolve().parent.parent + else: + root = Path.home() / ".openhands" / "workspaces" + + state_dir = root / "automation-state" + state_dir.mkdir(parents=True, exist_ok=True) + return str(state_dir / f"dd_monitor_{automation_id}.json") + + +def _default_since() -> str: + return ( + datetime.now(timezone.utc) - timedelta(minutes=INITIAL_LOOKBACK_MINUTES) + ).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def load_state(path: str) -> dict: + if os.path.exists(path): + try: + with open(path) as f: + return json.load(f) + except (json.JSONDecodeError, OSError) as exc: + print(f"Warning: state file unreadable ({exc}); starting fresh") + return { + "version": 1, + "last_poll_timestamp": _default_since(), + "active_conversation": None, + "known_patterns": {}, + } + + +def save_state(path: str, state: dict) -> None: + """Atomic write — write to .tmp then rename to avoid partial reads.""" + tmp = path + ".tmp" + with open(tmp, "w") as f: + json.dump(state, f, indent=2) + os.replace(tmp, path) + + +# ── Pattern archiving ────────────────────────────────────────────────────────── + +def _archive_file_path(state_path: str) -> str: + p = Path(state_path) + return str(p.parent / (p.stem + "_archive" + p.suffix)) + + +def archive_stale_patterns(state: dict, state_path: str) -> int: + """Move patterns last seen more than PATTERN_ARCHIVE_DAYS ago to a separate + archive file. Returns the number of patterns archived. + + The archive is a flat JSON object keyed by pattern UUID. Each entry gets + an ``archived_at`` timestamp added so old investigations can be correlated + with the time the pattern fell silent. + """ + cutoff = datetime.now(timezone.utc) - timedelta(days=PATTERN_ARCHIVE_DAYS) + + to_archive: dict = {} + to_keep: dict = {} + for pid, pattern in state.get("known_patterns", {}).items(): + last_seen_str = pattern.get("last_seen", "") + if not last_seen_str: + to_keep[pid] = pattern + continue + try: + last_seen = datetime.fromisoformat(last_seen_str.replace("Z", "+00:00")) + if last_seen < cutoff: + to_archive[pid] = pattern + else: + to_keep[pid] = pattern + except ValueError: + to_keep[pid] = pattern # keep if unparseable + + if not to_archive: + return 0 + + archive_path = _archive_file_path(state_path) + try: + with open(archive_path) as f: + archive: dict = json.load(f) + except (FileNotFoundError, json.JSONDecodeError, OSError): + archive = {} + + now_str = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + for pid, pattern in to_archive.items(): + archive[pid] = {**pattern, "archived_at": now_str} + + tmp = archive_path + ".tmp" + with open(tmp, "w") as f: + json.dump(archive, f, indent=2) + os.replace(tmp, archive_path) + + state["known_patterns"] = to_keep + names = [p.get("name", pid) for pid, p in to_archive.items()] + print(f"Archived {len(to_archive)} stale pattern(s) → {archive_path}: {names}") + return len(to_archive) + + +# ── Datadog API ──────────────────────────────────────────────────────────────── + +def query_dd_logs( + dd_api_key: str, + dd_app_key: str, + from_ts: str, + to_ts: str, +) -> list[dict]: + """Query Datadog logs and return a list of raw log event dicts (oldest first).""" + url = f"https://api.{DD_SITE}/api/v2/logs/events/search" + payload = json.dumps({ + "filter": {"query": DD_QUERY, "from": from_ts, "to": to_ts}, + "sort": "timestamp", + "page": {"limit": 1000}, + }).encode() + req = urllib.request.Request( + url, + data=payload, + headers={ + "DD-API-KEY": dd_api_key, + "DD-APPLICATION-KEY": dd_app_key, + "Content-Type": "application/json", + }, + method="POST", + ) + try: + with urllib.request.urlopen(req, timeout=30) as r: + return json.loads(r.read().decode()).get("data", []) + except urllib.error.HTTPError as exc: + body = exc.read().decode()[:500] + raise RuntimeError(f"Datadog API error {exc.code}: {body}") from exc + + +def _extract_message(log_event: dict) -> str: + """Extract a representative message string from a Datadog log event.""" + attrs = log_event.get("attributes", {}) + msg = ( + attrs.get("message") + or attrs.get("error", {}).get("message") + or attrs.get("msg") + or "" + ) + stack = attrs.get("error", {}).get("stack", "") + if stack and len(msg) < 200: + msg = msg + "\n" + stack[:200] + return msg[:MAX_LOG_MESSAGE_CHARS] + + +# ── Pattern matching ─────────────────────────────────────────────────────────── + +def match_log(message: str, known_patterns: dict) -> str | None: + """Return the ID of the first pattern that matches, or None.""" + for pattern_id, pattern in known_patterns.items(): + try: + if re.search(pattern["regex"], message, re.IGNORECASE | re.DOTALL): + return pattern_id + except re.error as exc: + print(f"Warning: invalid regex in pattern '{pattern.get('name')}': {exc}") + return None + + +# ── OpenHands conversation helpers ───────────────────────────────────────────── + +def _oh_request( + agent_url: str, api_key: str, method: str, path: str, body: dict | None = None +) -> dict: + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request( + f"{agent_url.rstrip('/')}{path}", + data=data, + headers={"X-Session-API-Key": api_key, "Content-Type": "application/json"}, + method=method, + ) + try: + with urllib.request.urlopen(req, timeout=30) as r: + return json.loads(r.read().decode()) + except urllib.error.HTTPError as exc: + raise RuntimeError( + f"Agent server {exc.code} on {method} {path}: {exc.read().decode()[:300]}" + ) from exc + + +def _fetch_settings(agent_url: str, api_key: str) -> dict: + try: + return _oh_request(agent_url, api_key, "GET", "/api/settings") + except Exception as exc: + print(f"Warning: could not fetch agent settings: {exc}") + return {} + + +def _get_agent_dict(agent_url: str, api_key: str) -> dict: + data = _fetch_settings(agent_url, api_key) + llm = data.get("agent_settings", {}).get("llm", {}) + return { + "kind": "Agent", + "llm": llm, + "tools": [{"name": "terminal"}, {"name": "file_editor"}], + } + + +def _get_mcp_config(agent_url: str, api_key: str) -> dict | None: + try: + data = _fetch_settings(agent_url, api_key) + mcp = data.get("agent_settings", {}).get("mcp_config") + if isinstance(mcp, dict) and mcp.get("mcpServers"): + return mcp + except Exception as exc: + print(f"Warning: could not fetch MCP config: {exc}") + return None + + +def _build_secrets_payload(agent_url: str, api_key: str) -> dict: + """Forward all user secrets to the spawned conversation as LookupSecret references.""" + try: + secrets_list = _oh_request(agent_url, api_key, "GET", "/api/settings/secrets").get("secrets", []) + except Exception as exc: + print(f"Warning: could not list secrets: {exc}") + return {} + return { + s["name"]: { + "kind": "LookupSecret", + "url": f"/api/settings/secrets/{s['name']}", + **( {"headers": {"X-Session-API-Key": api_key}} if api_key else {} ), + **( {"description": s["description"]} if s.get("description") else {} ), + } + for s in secrets_list + if s.get("name") + } + + +def create_conversation( + agent_url: str, api_key: str, initial_message: str, workspace_dir: str +) -> str: + """Create an OpenHands conversation and return its ID.""" + payload: dict = { + "workspace": {"working_dir": workspace_dir}, + "agent": _get_agent_dict(agent_url, api_key), + "initial_message": {"content": [{"text": initial_message}]}, + } + if secrets := _build_secrets_payload(agent_url, api_key): + payload["secrets"] = secrets + if mcp := _get_mcp_config(agent_url, api_key): + payload["mcp_config"] = mcp + return _oh_request(agent_url, api_key, "POST", "/api/conversations", payload)["id"] + + +def conversation_status(agent_url: str, api_key: str, conv_id: str) -> str: + return _oh_request(agent_url, api_key, "GET", f"/api/conversations/{conv_id}").get( + "execution_status", "unknown" + ) + + +# ── Spike detection ──────────────────────────────────────────────────────────── + +def _is_spike(current_count: int, history_before_this_run: list[int]) -> bool: + """Return True if current_count significantly exceeds the recent baseline. + + Requires MIN_RUNS_FOR_SPIKE history entries before activating. + When all recent counts are zero, any non-zero value is a spike. + """ + if not history_before_this_run: + return False + if len(history_before_this_run) < MIN_RUNS_FOR_SPIKE: + return False + recent = history_before_this_run[-MIN_RUNS_FOR_SPIKE:] + baseline = sum(recent) / len(recent) + if baseline == 0: + return current_count > 0 + return current_count > baseline * SPIKE_MULTIPLIER + + +# ── Investigation prompt ─────────────────────────────────────────────────────── + +def _build_prompt( + state_file_path: str, + from_ts: str, + to_ts: str, + unknown_samples: list[str], + total_unknown: int, + spiking: list[tuple[str, dict, int]], +) -> str: + lines = [ + "# Datadog Error Monitor — Investigation Request", + "", + "## Context", + f"- **Datadog query:** `{DD_QUERY}`", + f"- **Time window:** {from_ts} → {to_ts}", + f"- **State file:** `{state_file_path}`", + f"- **Archive file:** `{_archive_file_path(state_file_path)}`", + "", + "## Investigation Budget", + "", + f"You have **{INVESTIGATION_BUDGET} tool calls** (terminal commands, file reads,", + "Datadog API queries combined) to spend across **all tasks**.", + "", + "- Simple cases (clear stack trace → matching code location): 2–3 calls", + "- Ambiguous cases: spend at most 3–4 calls, then **declare inconclusive**", + "- Stop when the budget is exhausted, even if patterns remain uninvestigated", + "", + "If you cannot identify the root cause within your allocated calls, set the", + "pattern's `description` to `\"Inconclusive — \"`", + "and move on. An inconclusive finding is more useful than no finding.", + "", + "## Tasks", + "", + "Work through the following tasks in order. The state file is a JSON document;", + "read it, make your changes, then write it back **atomically** (write to a `.tmp`", + "file, then `os.replace`). Preserve all existing top-level fields.", + "", + "---", + "", + "### Task 1 — Categorize unknown logs", + ] + + if total_unknown == 0: + lines += ["", "No uncategorized logs this run. Proceed to Task 2."] + else: + truncation_note = ( + f" Only the first {MAX_UNKNOWN_LOGS} of {total_unknown} are shown." + if total_unknown > MAX_UNKNOWN_LOGS else "" + ) + lines += [ + "", + f"**{total_unknown} log event(s)** did not match any known pattern.{truncation_note}", + "", + "**Before creating any new pattern**, read `known_patterns` from the state file", + "and check for overlap with the samples below:", + "", + "- Test each existing pattern's `regex` against the new samples:", + " `re.search(pattern['regex'], sample, re.IGNORECASE | re.DOTALL)`", + "- If an existing regex already covers a sample, do **not** create a new pattern;", + " note the overlap in the Slack summary (Task 4) instead.", + "- If root causes appear similar but error messages are distinct, create a new", + " pattern and cross-reference both `description` fields.", + "", + "For each genuinely new error class, add an entry to `known_patterns` using a", + "new UUID key (`import uuid; str(uuid.uuid4())`).", + "", + "Each new pattern requires these fields:", + "```json", + "{", + ' "name": "Concise human-readable label (e.g. Redis connection timeout)",', + ' "regex": "Python regex — re.IGNORECASE | re.DOTALL — matching this error class",', + ' "first_seen": "",', + ' "last_seen": "",', + ' "total_events": ,', + ' "description": "Brief notes: likely cause, related service, stack trace clues.",', + ' "run_history": [],', + ' "examples": [{"timestamp": "...", "message": "..."}]', + "}", + "```", + "", + "**Regex quality rules:**", + "- Match the error *class*, not a single occurrence", + "- Avoid embedding timestamps, request IDs, memory addresses, or UUIDs", + "- Use `.*` to skip variable parts between fixed anchor strings", + "- Prefer fewer, broader patterns over many narrow ones", + "", + "", + ] + for i, msg in enumerate(unknown_samples, 1): + lines.append(f"[{i}] {msg}") + lines.append("") + + # ── Task 2: Deployment correlation ──────────────────────────────────────── + lines += [ + "", + "---", + "", + "### Task 2 — Correlate errors with deployments", + "", + "For every newly created pattern (Task 1) and every spiking pattern (listed in", + "Task 3), compare `first_seen` against recent deployments to identify likely cause.", + "", + ] + + if REPO_CONFIGS: + lines += [ + "Fetch tags and list recent ones with dates for each configured repository:", + "```bash", + ] + for cfg in REPO_CONFIGS: + repo_label = cfg.get("remote", cfg["path"]) + lines += [ + f"# {repo_label}", + f"git -C {cfg['path']} fetch --tags --quiet origin 2>/dev/null", + f"git -C {cfg['path']} log --tags --simplify-by-decoration" + " --pretty=format:'%D %ai' --since='60 days ago' | grep 'tag:'", + ] + lines += [ + "```", + "", + "> **Note:** This step assumes git tags mark production deployments.", + "> If your team deploys differently (branch tip, CI artefact, Datadog", + "> deployment markers), confirm the right signal with the repo owner and", + "> adjust this step accordingly.", + "", + "If a pattern's `first_seen` falls within a few hours after a tag, update its", + "`description` to note the correlation, e.g.:", + '`"Errors first seen ~2 h after deploy v2.3.1 (2024-03-12T14:05Z). Likely introduced then."`', + ] + else: + lines += ["*(No repositories configured — skip deployment correlation.)*"] + + # ── Task 3: Investigate spiking patterns ────────────────────────────────── + lines += [ + "", + "---", + "", + "### Task 3 — Investigate spiking patterns", + "", + ] + + if not spiking: + lines += ["No patterns have spiked this run. Proceed to Task 4."] + else: + lines += [ + "The following patterns have counts significantly above their recent baseline.", + "", + "For **each** spiking pattern, work through these steps within your budget:", + "", + "1. `git -C pull --ff-only origin` to ensure you are on the latest code", + "2. Check the stack traces in the examples for a specific code location", + "3. If a location is identified: read that file and check recent changes", + " (`git -C log -8 --oneline -- `)", + "4. If the cause is still unclear, run **one** follow-up Datadog query:", + " ```bash", + f' curl -s -X POST "https://api.{DD_SITE}/api/v2/logs/events/search" \\', + ' -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \\', + ' -H "Content-Type: application/json" \\', + " -d '{\"filter\":{\"query\":\"\"," + "\"from\":\"now-1h\",\"to\":\"now\"},\"page\":{\"limit\":10}}'", + " ```", + "5. If still unclear after step 4: **declare inconclusive** and move on", + "", + "After investigating each pattern, **overwrite its `description`** in the state", + "file with your current findings.", + "", + ] + + for pid, pattern, count in spiking: + history = pattern.get("run_history", []) + recent = history[-MIN_RUNS_FOR_SPIKE:] if len(history) >= MIN_RUNS_FOR_SPIKE else history + baseline_str = f"{sum(recent) / len(recent):.1f}" if recent else "N/A" + lines += [ + f"#### `{pattern['name']}`", + f"- Count this run: **{count}** | Baseline: {baseline_str}" + f" (last {len(recent)} runs: `{recent}`)", + f"- First seen: {pattern.get('first_seen', 'unknown')}", + f"- Last seen: {pattern.get('last_seen', 'unknown')}", + f"- Total events: {pattern.get('total_events', 'unknown')}", + f"- Regex: `{pattern.get('regex', 'N/A')}`", + f"- Description: {pattern.get('description', '(none yet)')}", + "", + ] + examples = pattern.get("examples", [])[:EXAMPLES_PER_PATTERN] + if examples: + lines.append("Recent examples:") + for ex in examples: + lines.append(f" - `{ex.get('message', '')[:300]}`") + lines.append("") + + lines += [ + "**If you identify a code-level fix:**", + "- Create a focused PR using the appropriate git host (GitHub / GitLab / Bitbucket)", + "- Only open a PR when you are **highly confident** the fix is correct", + "- Include a description referencing the error pattern name and `first_seen`", + "", + "**Otherwise** (infrastructure / config / data issue, or inconclusive):", + "- Do NOT create a PR", + "- Capture your findings in the pattern's `description` for the Slack summary", + ] + + if REPO_CONFIGS: + lines += [ + "", + "**Repositories (already cloned on this machine):**", + ] + for cfg in REPO_CONFIGS: + lines.append( + f"- `{cfg['path']}` — {cfg.get('host', 'git')} · `{cfg.get('remote', '')}`" + ) + + # ── Task 4: Slack summary ────────────────────────────────────────────────── + lines += [ + "", + "---", + "", + "### Task 4 — Post Slack summary", + "", + f"Post to Slack channel `{SLACK_CHANNEL_ID}` using the `SLACK_BOT_TOKEN` secret.", + "", + "**Only post if there were actual findings** (new patterns, spikes, or deployment", + "correlations). Skip this task entirely if nothing triggered.", + "", + "Include:", + "- New error patterns: name, event count, and brief description", + "- Spiking patterns: diagnosis, deployment correlation if found, PR link if opened", + "- Inconclusive patterns: name + what was tried — flag for manual investigation", + "", + "Keep it concise — bullet points preferred. Example:", + "", + "```", + f"🔴 *Datadog Error Monitor — {from_ts}*", + "", + "New patterns (2):", + " • Redis timeout in CacheService — 12 events, likely from deploy v2.3.1", + " • JWT validation failure — 4 events — inconclusive, needs manual review", + "Spike:", + " • NullPointerException in PaymentService — 47 events (baseline ~3)", + " → null check missing after optional-field refactor → PR #123 opened", + "```", + "", + "```bash", + "curl -s -X POST https://slack.com/api/chat.postMessage \\", + ' -H "Authorization: Bearer $SLACK_BOT_TOKEN" \\', + ' -H "Content-Type: application/json" \\', + f' -d \'{{\"channel\": \"{SLACK_CHANNEL_ID}\", \"text\": \"YOUR SUMMARY\"}}\' \\', + " | python3 -c \"import json,sys; d=json.load(sys.stdin);" + " print('OK' if d.get('ok') else d.get('error'))\"", + "```", + ] + + return "\n".join(lines) + + +# ── Main ─────────────────────────────────────────────────────────────────────── + +def main() -> str | None: + state_path = _state_file_path() + print(f"State file: {state_path}") + state = load_state(state_path) + + # ── Archive patterns not seen recently ─────────────────────────────────── + archive_stale_patterns(state, state_path) + + # ── Resolve Datadog secrets ────────────────────────────────────────────── + try: + dd_api_key = get_secret("DD_API_KEY") + dd_app_key = get_secret("DD_APP_KEY") + except Exception as exc: + raise RuntimeError(f"Failed to fetch Datadog secrets: {exc}") from exc + + agent_url = os.environ.get("AGENT_SERVER_URL", "").rstrip("/") + api_key = _env_api_key() + + # ── Define query window with overlap ───────────────────────────────────── + now = datetime.now(timezone.utc) + to_ts = (now - timedelta(seconds=10)).strftime("%Y-%m-%dT%H:%M:%SZ") + from_dt = ( + datetime.fromisoformat(state["last_poll_timestamp"].replace("Z", "+00:00")) + - timedelta(seconds=OVERLAP_SECONDS) + ) + from_ts = from_dt.strftime("%Y-%m-%dT%H:%M:%SZ") + + print(f"Querying: {DD_QUERY!r} [{from_ts} → {to_ts}]") + + # ── Query Datadog ──────────────────────────────────────────────────────── + logs = query_dd_logs(dd_api_key, dd_app_key, from_ts, to_ts) + print(f"Retrieved {len(logs)} log events") + + # ── Match logs against known patterns ──────────────────────────────────── + known_patterns = state.setdefault("known_patterns", {}) + pattern_counts: dict[str, int] = {} + unknown_samples: list[str] = [] + total_unknown = 0 + + for log_event in logs: + msg = _extract_message(log_event) + pid = match_log(msg, known_patterns) + if pid: + pattern_counts[pid] = pattern_counts.get(pid, 0) + 1 + pattern = known_patterns[pid] + pattern["last_seen"] = now.strftime("%Y-%m-%dT%H:%M:%SZ") + pattern["total_events"] = pattern.get("total_events", 0) + 1 + pattern.setdefault("examples", []) + new_example = { + "timestamp": log_event.get("attributes", {}).get("timestamp", ""), + "message": msg, + } + pattern["examples"] = ([new_example] + pattern["examples"])[:EXAMPLES_PER_PATTERN] + else: + total_unknown += 1 + if len(unknown_samples) < MAX_UNKNOWN_LOGS: + unknown_samples.append(msg) + + # ── Update run_history and detect spikes ───────────────────────────────── + # History is updated AFTER spike detection so we compare against prior runs. + spiking: list[tuple[str, dict, int]] = [] + for pid, pattern in known_patterns.items(): + history = pattern.setdefault("run_history", []) + count = pattern_counts.get(pid, 0) + if _is_spike(count, history): + spiking.append((pid, pattern, count)) + print(f"Spike: '{pattern['name']}' — {count} events (history: {history[-MIN_RUNS_FOR_SPIKE:]})") + history.append(count) + pattern["run_history"] = history[-MAX_RUN_HISTORY:] + + if pattern_counts: + print("Pattern counts:", {known_patterns[pid]["name"]: c for pid, c in pattern_counts.items()}) + print(f"Unknown: {total_unknown} total, {len(unknown_samples)} sampled") + + # ── Check active conversation ───────────────────────────────────────────── + active = state.get("active_conversation") + conversation_id: str | None = None + + if active and agent_url: + conv_id = active["id"] + try: + status = conversation_status(agent_url, api_key, conv_id) + except Exception as exc: + print(f"Warning: could not check conversation {conv_id}: {exc}") + status = "unknown" + + print(f"Active conversation {conv_id} → status={status}") + + # Treat conversations stuck in a non-terminal state beyond the timeout as stuck. + if status not in ("idle", "finished", "error", "stuck"): + started_at_str = active.get("started_at", "") + if started_at_str: + try: + started_dt = datetime.fromisoformat(started_at_str.replace("Z", "+00:00")) + elapsed_min = (now - started_dt).total_seconds() / 60 + if elapsed_min > STUCK_CONVERSATION_MINUTES: + print( + f"Conversation {conv_id} has been running for {elapsed_min:.0f} min " + f"(> {STUCK_CONVERSATION_MINUTES} min limit) — treating as stuck" + ) + status = "stuck" + except ValueError: + pass + + if status in ("idle", "finished", "error", "stuck"): + print("Conversation finished — clearing active slot") + state["active_conversation"] = None + active = None + else: + # Investigation still running — save updated pattern data and exit + state["last_poll_timestamp"] = to_ts + save_state(state_path, state) + print("Investigation in progress — skipping trigger evaluation") + return conv_id + + # ── Evaluate triggers ──────────────────────────────────────────────────── + should_trigger = total_unknown > 0 or bool(spiking) + + if should_trigger and agent_url: + # Use the first configured repo as workspace so the agent has code access + if REPO_CONFIGS: + workspace_dir = REPO_CONFIGS[0]["path"] + else: + workspace_base = os.environ.get("WORKSPACE_BASE", "") + root = ( + str(Path(workspace_base).resolve().parent.parent) + if workspace_base + else os.path.expanduser("~/.openhands/workspaces") + ) + workspace_dir = os.path.join(root, "dd-monitor-investigations") + os.makedirs(workspace_dir, exist_ok=True) + + prompt = _build_prompt( + state_file_path=state_path, + from_ts=from_ts, + to_ts=to_ts, + unknown_samples=unknown_samples, + total_unknown=total_unknown, + spiking=spiking, + ) + + trigger_parts = [] + if total_unknown > 0: + trigger_parts.append(f"{total_unknown} unknown logs") + if spiking: + trigger_parts.append("spikes in: " + ", ".join(p["name"] for _, p, _ in spiking)) + trigger_summary = "; ".join(trigger_parts) + + try: + conversation_id = create_conversation(agent_url, api_key, prompt, workspace_dir) + print(f"Started investigation conversation: {conversation_id}") + state["active_conversation"] = { + "id": conversation_id, + "started_at": now.strftime("%Y-%m-%dT%H:%M:%SZ"), + "trigger_summary": trigger_summary, + "status": "running", + } + except Exception as exc: + print(f"Failed to create conversation (will retry next run): {exc}") + elif should_trigger: + print("Triggers detected but AGENT_SERVER_URL not set — cannot create conversation") + else: + print("No triggers — no investigation needed") + + # ── Save state and return ───────────────────────────────────────────────── + state["last_poll_timestamp"] = to_ts + save_state(state_path, state) + print(f"State saved. Next poll window starts from: {to_ts}") + return conversation_id + + +if __name__ == "__main__": + try: + conv_id = main() + fire_callback("COMPLETED", conversation_id=conv_id) + except Exception as exc: + print(f"Fatal error: {exc}", file=sys.stderr) + fire_callback("FAILED", error=str(exc)) + sys.exit(1) diff --git a/skills/index.js b/skills/index.js index 0737c2a2..c9d77943 100644 --- a/skills/index.js +++ b/skills/index.js @@ -86,6 +86,14 @@ export const SKILLS_CATALOG = [ ], "content": "# Datadog\n\n\nBefore performing any Datadog operations, first check if the required environment variables are set:\n\n```bash\n[ -n \"$DD_API_KEY\" ] && echo \"DD_API_KEY is set\" || echo \"DD_API_KEY is NOT set\"\n[ -n \"$DD_APP_KEY\" ] && echo \"DD_APP_KEY is set\" || echo \"DD_APP_KEY is NOT set\"\n[ -n \"$DD_SITE\" ] && echo \"DD_SITE is set\" || echo \"DD_SITE is NOT set\"\n```\n\nIf any of these variables are missing, ask the user to provide them before proceeding:\n- **DD_API_KEY**: Datadog API key\n- **DD_APP_KEY**: Datadog Application key\n- **DD_SITE**: Datadog site (e.g., `datadoghq.com`, `datadoghq.eu`, `us3.datadoghq.com`)\n\n\n## Authentication Headers\n\n```bash\n-H \"DD-API-KEY: ${DD_API_KEY}\" \\\n-H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\" \\\n-H \"Content-Type: application/json\"\n```\n\n## Query Logs\n\n```bash\ncurl -s -X POST \"https://api.${DD_SITE}/api/v2/logs/events/search\" \\\n -H \"DD-API-KEY: ${DD_API_KEY}\" \\\n -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"filter\": {\n \"query\": \"service:my-service status:error\",\n \"from\": \"now-1h\",\n \"to\": \"now\"\n },\n \"sort\": \"-timestamp\",\n \"page\": {\"limit\": 50}\n }' | jq .\n```\n\n## Query Metrics\n\n```bash\ncurl -s -G \"https://api.${DD_SITE}/api/v1/query\" \\\n -H \"DD-API-KEY: ${DD_API_KEY}\" \\\n -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\" \\\n --data-urlencode \"query=avg:system.cpu.user{*}\" \\\n --data-urlencode \"from=$(date -d '1 hour ago' +%s)\" \\\n --data-urlencode \"to=$(date +%s)\" | jq .\n```\n\n## Query APM Traces\n\n```bash\ncurl -s -X POST \"https://api.${DD_SITE}/api/v2/spans/events/search\" \\\n -H \"DD-API-KEY: ${DD_API_KEY}\" \\\n -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"filter\": {\n \"query\": \"service:my-service\",\n \"from\": \"now-1h\",\n \"to\": \"now\"\n },\n \"sort\": \"-timestamp\",\n \"page\": {\"limit\": 25}\n }' | jq .\n```\n\n## List Monitors\n\n```bash\ncurl -s -G \"https://api.${DD_SITE}/api/v1/monitor\" \\\n -H \"DD-API-KEY: ${DD_API_KEY}\" \\\n -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\" | jq .\n```\n\n## Documentation\n\n- [Logs API](https://docs.datadoghq.com/api/latest/logs/)\n- [Metrics API](https://docs.datadoghq.com/api/latest/metrics/)\n- [APM/Tracing API](https://docs.datadoghq.com/api/latest/tracing/)\n- [Monitors API](https://docs.datadoghq.com/api/latest/monitors/)\n- [Events API](https://docs.datadoghq.com/api/latest/events/)\n- [Dashboards API](https://docs.datadoghq.com/api/latest/dashboards/)" }, + { + "name": "datadog-error-monitor", + "description": "This skill should be used when the user asks to \"monitor Datadog for errors\", \"watch Datadog logs for error spikes\", \"investigate Datadog errors automatically\", \"alert on new or recurring errors in Datadog\", or \"set up automated error monitoring with Datadog and Slack\". Guides the user through creating a cron automation that polls Datadog logs every 15 minutes, maintains a library of known error patterns, and triggers an OpenHands investigation conversation when new or spiking errors are detected. The investigation agent categorizes unknown errors, investigates root causes in local codebases, optionally creates PRs, and posts a concise summary to Slack.", + "triggers": [ + "/datadog-monitor:setup" + ], + "content": "# Datadog Error Monitor\n\nCreate a cron automation that polls Datadog logs every 15 minutes.\n\nOn each run the script:\n1. Queries Datadog using a configured log filter (e.g. `service:(api OR worker) status:error`)\n2. Matches log events against a library of known error patterns (regex-based, stored in a state file)\n3. Tracks hit counts per pattern across runs\n4. When **new/uncategorized logs** or a **significant count spike** is detected,\n starts a single OpenHands investigation conversation\n\nWhen an investigation conversation runs it:\n- Categorizes uncategorized logs into named patterns and writes them to the state file\n- Investigates root causes in the configured local codebases\n- Creates a PR only when highly confident of a code-level fix\n- Posts a concise summary to the configured Slack channel\n\n> **Local mode only.** Monitored code repositories must be cloned on the same\n> machine running OpenHands.\n\n---\n\n## Prerequisites\n\n### Required secrets\n\nVerify all of the following are set in **OpenHands Settings → Secrets**:\n\n| Secret name | Description |\n|---|---|\n| `DD_API_KEY` | Datadog API key |\n| `DD_APP_KEY` | Datadog Application key (required for log search — distinct from the API key) |\n| `SLACK_BOT_TOKEN` | Slack bot token with `chat:write` scope |\n\nCheck with:\n```bash\n[ -n \"$DD_API_KEY\" ] && echo \"DD_API_KEY is set\" || echo \"DD_API_KEY is NOT set\"\n[ -n \"$DD_APP_KEY\" ] && echo \"DD_APP_KEY is set\" || echo \"DD_APP_KEY is NOT set\"\n[ -n \"$SLACK_BOT_TOKEN\" ] && echo \"SLACK_BOT_TOKEN is set\" || echo \"SLACK_BOT_TOKEN is NOT set\"\n```\n\nIf any are missing, inform the user and stop — the automation cannot function without them.\n\nGit host token(s) are confirmed per-repository in Step 5.\n\n### Optional secret\n\n| Secret name | Default | Purpose |\n|---|---|---|\n| `OPENHANDS_URL` | `http://localhost:8000` | Base URL for conversation links |\n\n---\n\n## Setup Workflow\n\nFollow these steps in order.\n\n### Step 1 — Verify Datadog credentials\n\n```bash\nDD_SITE_TMP=\"datadoghq.com\"\ncurl -s -X POST \"https://api.${DD_SITE_TMP}/api/v2/logs/events/search\" \\\n -H \"DD-API-KEY: $DD_API_KEY\" \\\n -H \"DD-APPLICATION-KEY: $DD_APP_KEY\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"filter\":{\"query\":\"*\",\"from\":\"now-1m\",\"to\":\"now\"},\"page\":{\"limit\":1}}' \\\n | python3 -c \"\nimport json, sys\nd = json.load(sys.stdin)\nif 'errors' in d:\n print('ERROR:', d['errors'])\nelif 'data' in d:\n print('OK — credentials valid, got', len(d['data']), 'sample event(s)')\nelse:\n print('Unexpected response:', list(d.keys()))\n\"\n```\n\n- If `DD_API_KEY` or `DD_APP_KEY` is missing → tell the user and stop.\n- If the API returns a 403 → tell the user to check both keys in Datadog:\n *Organization Settings → API Keys* (for `DD_API_KEY`) and\n *Organization Settings → Application Keys* (for `DD_APP_KEY`).\n\n### Step 2 — Confirm Datadog site\n\nAsk: *\"What is your Datadog site?*\n*(Press Enter for the default: `datadoghq.com`)*\n*Other options: `datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`\"*\n\nRecord as `DD_SITE`. Default: `\"datadoghq.com\"`.\n\nRe-run the Step 1 credential check substituting the confirmed site.\n\n### Step 3 — Collect the Datadog log query\n\nAsk: *\"What Datadog log query should be monitored? This filter runs every 15 minutes.*\n*Example: `service:(deploy OR runtime-api) status:error`*\n*Example: `service:payment-api (@http.status_code:5* OR status:error)`*\n*Tip: develop and test your query in the Datadog Logs Explorer first.\"*\n\nRun a test query to show the user what they will be monitoring:\n```bash\nUSER_QUERY=\"REPLACE_WITH_USER_QUERY\"\nDD_SITE=\"REPLACE_WITH_DD_SITE\"\ncurl -s -X POST \"https://api.${DD_SITE}/api/v2/logs/events/search\" \\\n -H \"DD-API-KEY: $DD_API_KEY\" \\\n -H \"DD-APPLICATION-KEY: $DD_APP_KEY\" \\\n -H \"Content-Type: application/json\" \\\n -d \"{\n \\\"filter\\\": {\\\"query\\\": \\\"${USER_QUERY}\\\", \\\"from\\\": \\\"now-15m\\\", \\\"to\\\": \\\"now\\\"},\n \\\"sort\\\": \\\"-timestamp\\\",\n \\\"page\\\": {\\\"limit\\\": 5}\n }\" | python3 -c \"\nimport json, sys\nd = json.load(sys.stdin)\nif 'errors' in d:\n print('ERROR:', d['errors'])\n sys.exit(1)\nevents = d.get('data', [])\nprint(f'{len(events)} event(s) in the last 15 minutes (showing up to 5):')\nfor e in events:\n attrs = e.get('attributes', {})\n ts = attrs.get('timestamp', '')\n msg = (attrs.get('message') or attrs.get('error', {}).get('message') or '')[:120]\n svc = attrs.get('service', '')\n print(f' [{ts}] [{svc}] {msg}')\n\"\n```\n\nShow the output to the user. If the query looks correct, record as `DD_QUERY`.\nIf they want to adjust it, iterate until they are satisfied.\n\n### Step 4 — Collect Slack configuration\n\nAsk: *\"Which Slack channel should investigation summaries be posted to?*\n*Provide the channel name (e.g. `#errors-alerts`) or ID (e.g. `C0123456789`).\"*\n\nVerify the bot token:\n```bash\ncurl -s https://slack.com/api/auth.test \\\n -H \"Authorization: Bearer $SLACK_BOT_TOKEN\" \\\n | python3 -c \"import json,sys; d=json.load(sys.stdin); print('Bot:', d.get('user')) if d.get('ok') else print('ERROR:', d.get('error'))\"\n```\n\nIf the user provided a channel name, resolve it to an ID:\n```bash\nCHANNEL_NAME=\"REPLACE_WITH_NAME_WITHOUT_HASH\"\ncurl -s \"https://slack.com/api/conversations.list?types=public_channel,private_channel&limit=200&exclude_archived=true\" \\\n -H \"Authorization: Bearer $SLACK_BOT_TOKEN\" \\\n | python3 -c \"\nimport json, sys\ndata = json.load(sys.stdin)\nif not data.get('ok'):\n print('ERROR:', data.get('error'))\n sys.exit(1)\nfor ch in data.get('channels', []):\n if ch['name'] == '${CHANNEL_NAME}'.lstrip('#'):\n print(ch['id'])\n break\nelse:\n print('Channel not found — provide the channel ID directly (right-click channel → Copy link)')\n\"\n```\n\nRecord as `SLACK_CHANNEL_ID`.\n\n### Step 5 — Collect monitored repositories\n\nAsk: *\"Which local code repositories should the investigation agent have access to?*\n*These must already be cloned on this machine.*\n*For each, provide the absolute path and the git host (GitHub / GitLab / Bitbucket).*\n*Example: `/home/user/projects/my-api` (GitHub)\"*\n\nFor each repository:\n\n**a) Verify the path is a git repo:**\n```bash\ncd /path/to/repo && git remote -v && git log --oneline -1\n```\n\n**b) Confirm the git host token is available:**\n\n| Host | Required secret | Verification |\n|---|---|---|\n| GitHub | `GITHUB_PERSONAL_ACCESS_TOKEN` | `curl -s https://api.github.com/user -H \"Authorization: Bearer $GITHUB_PERSONAL_ACCESS_TOKEN\" \\| python3 -c \"import json,sys; d=json.load(sys.stdin); print(d.get('login', d.get('message')))\"` |\n| GitLab | `GITLAB_TOKEN` | `curl -s https://gitlab.com/api/v4/user -H \"PRIVATE-TOKEN: $GITLAB_TOKEN\" \\| python3 -c \"import json,sys; d=json.load(sys.stdin); print(d.get('username', d.get('message')))\"` |\n| Bitbucket | `BITBUCKET_TOKEN` | `curl -s https://api.bitbucket.org/2.0/user --user \"$BITBUCKET_TOKEN\" \\| python3 -c \"import json,sys; d=json.load(sys.stdin); print(d.get('username', d.get('error')))\"` |\n\nIf the required token is missing, inform the user. The investigation agent\ncan still investigate code but will not be able to create PRs until the token\nis added to OpenHands Settings → Secrets.\n\n**c) Extract the remote identifier** from `git remote get-url origin`.\n\nBuild `REPO_CONFIGS` as a Python list literal:\n```python\nREPO_CONFIGS: list[dict] = [\n {\"path\": \"/path/to/repo1\", \"host\": \"github\", \"remote\": \"owner/repo1\"},\n {\"path\": \"/path/to/repo2\", \"host\": \"gitlab\", \"remote\": \"owner/repo2\"},\n {\"path\": \"/path/to/repo3\", \"host\": \"bitbucket\", \"remote\": \"owner/repo3\"},\n]\n```\n\n### Step 6 — Tune parameters\n\nAsk about the following settings. Offer the defaults and accept Enter to keep them:\n\n| Parameter | Default | Prompt |\n|---|---|---|\n| `MAX_UNKNOWN_LOGS` | `100` | *\"Max uncategorized log samples to send to the agent per run?\"* |\n| `EXAMPLES_PER_PATTERN` | `3` | *\"How many example logs to keep per error pattern?\"* |\n| `SPIKE_MULTIPLIER` | `3.0` | *\"Spike threshold — alert when count exceeds this multiple of the rolling 3-run average?\"* |\n\n### Step 7 — Dry run confirmation\n\nRun the Step 3 query one final time with a 15-minute window and show the user:\n- Total number of matching events\n- The first 3 example log messages\n\nAsk: *\"This is what the automation will monitor every 15 minutes. Does it look right?*\n*Shall I create the automation?\"*\n\nIf the user wants changes, return to Step 3.\n\n### Step 8 — Generate the automation script\n\nRead `scripts/main.py` from this skill's directory and **copy it verbatim**.\nApply exactly the following constant substitutions near the top of the file:\n\n> **Do not reimplement or hand-write a replacement script.** Only substitute the\n> configuration constants below — all other logic must remain unchanged.\n\n| Placeholder | Replace with |\n|---|---|\n| `DD_QUERY = \"service:(deploy OR runtime-api) status:error\"` | `DD_QUERY = \"{dd_query}\"` |\n| `DD_SITE = \"datadoghq.com\"` | `DD_SITE = \"{dd_site}\"` |\n| `SLACK_CHANNEL_ID = \"C0123456789\"` | `SLACK_CHANNEL_ID = \"{channel_id}\"` |\n| `REPO_CONFIGS: list[dict] = []` | `REPO_CONFIGS: list[dict] = {repo_configs_literal}` |\n| `MAX_UNKNOWN_LOGS = 100` | `MAX_UNKNOWN_LOGS = {max_unknown}` |\n| `EXAMPLES_PER_PATTERN = 3` | `EXAMPLES_PER_PATTERN = {examples_per}` |\n| `SPIKE_MULTIPLIER = 3.0` | `SPIKE_MULTIPLIER = {multiplier}` |\n| `DEFAULT_OPENHANDS_URL = \"http://localhost:8000\"` | `DEFAULT_OPENHANDS_URL = \"{url}\"` (keep default if user has no preference) |\n\nWrite the script to a temporary build directory:\n```bash\nmkdir -p /tmp/dd-monitor-build\n# copy and substitute scripts/main.py → /tmp/dd-monitor-build/main.py\n```\n\nValidate syntax and confirm substitutions:\n```bash\npython3 -m py_compile /tmp/dd-monitor-build/main.py && echo \"Syntax OK\"\ngrep -n 'DD_QUERY = ' /tmp/dd-monitor-build/main.py\ngrep -n 'DD_SITE = ' /tmp/dd-monitor-build/main.py\ngrep -n 'SLACK_CHANNEL_ID = ' /tmp/dd-monitor-build/main.py\ngrep -n 'REPO_CONFIGS' /tmp/dd-monitor-build/main.py\ngrep -n 'def get_secret' /tmp/dd-monitor-build/main.py\ngrep -n 'def main' /tmp/dd-monitor-build/main.py\n```\n\nIf any check fails, re-copy the template and redo the substitutions.\n\n### Step 9 — Package and upload\n\nDetermine the Automation backend URL from the `` block in\nyour system context.\n\n```bash\ntar -czf /tmp/dd-monitor.tar.gz -C /tmp/dd-monitor-build .\n\nOPENHANDS_HOST=\"\"\n\nTARBALL_PATH=$(curl -s -X POST \\\n \"${OPENHANDS_HOST}/api/automation/v1/uploads?name=datadog-error-monitor\" \\\n -H \"X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY\" \\\n -H \"Content-Type: application/gzip\" \\\n --data-binary @/tmp/dd-monitor.tar.gz \\\n | python3 -c \"import json,sys; print(json.load(sys.stdin)['tarball_path'])\")\n\necho \"Uploaded: $TARBALL_PATH\"\n```\n\n### Step 10 — Create the automation\n\n```bash\ncurl -s -X POST \"${OPENHANDS_HOST}/api/automation/v1\" \\\n -H \"X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY\" \\\n -H \"Content-Type: application/json\" \\\n -d \"{\n \\\"name\\\": \\\"Datadog Error Monitor\\\",\n \\\"trigger\\\": {\\\"type\\\": \\\"cron\\\", \\\"schedule\\\": \\\"*/15 * * * *\\\"},\n \\\"tarball_path\\\": \\\"$TARBALL_PATH\\\",\n \\\"entrypoint\\\": \\\"python3 main.py\\\",\n \\\"timeout\\\": 120\n }\" | python3 -m json.tool\n```\n\nRecord the returned `id`.\n\n### Step 11 — Confirm and hand off\n\nTell the user:\n\n> ✅ **Datadog Error Monitor** is running!\n>\n> - **Automation ID:** `{id}`\n> - **Query:** `{dd_query}`\n> - **Site:** `{dd_site}`\n> - **Alert channel:** `{slack_channel}`\n> - **Repos:** `{repo_paths}`\n> - **Schedule:** every 15 minutes (`*/15 * * * *`)\n> - **State file:** `~/.openhands/workspaces/automation-state/dd_monitor_{id}.json`\n>\n> **What happens next:**\n> The first run will treat all matching logs as uncategorized and start an\n> investigation conversation. The agent will build the initial pattern library.\n> After a few runs, the system converges on a stable baseline and only alerts\n> on new error types or significant spikes.\n>\n> You can inspect or edit the state file at any time to adjust patterns, remove\n> stale ones, or reset the system entirely by deleting the file.\n\n---\n\n## Runtime Behaviour (per 15-min poll)\n\nEach cron run executes `main.py` with a 120-second timeout:\n\n1. **Loads state** — picks up any pattern updates written by the previous agent run\n2. **Queries Datadog** — window: `[last_poll − 60 s overlap, now − 10 s]`\n3. **Matches logs** — each event tested against `known_patterns` regexes\n4. **Updates history** — appends counts to each pattern's `run_history` (capped at 20); refreshes `examples` and `last_seen`\n5. **Checks active conversation** — if one is running, saves updated state and exits; spike evaluation is skipped until the investigation finishes\n6. **Evaluates triggers:**\n - Unknown logs (count ≥ 1) → trigger\n - Pattern spike (`current > mean(last 3 runs) × SPIKE_MULTIPLIER`) → trigger\n7. **If triggered:** builds the investigation prompt and starts one OpenHands conversation; records its ID in state\n8. **Saves state** and fires the completion callback\n\nThe investigation agent (asynchronous):\n- Categorizes unknown logs into named patterns → writes back to state file\n- Investigates spiking patterns in the configured repos\n- Creates PRs only if highly confident of a code-level fix\n- Posts a Slack summary with findings\n\n---\n\n## Troubleshooting\n\n| Symptom | Likely cause | Fix |\n|---|---|---|\n| `Datadog API error 403` | `DD_APP_KEY` invalid or wrong site | Verify both keys in Datadog Org Settings; re-check `DD_SITE` |\n| No events returned | Query too narrow, wrong site, or no recent errors | Test query in Datadog Logs Explorer UI |\n| Agent creates very specific patterns | LLM being over-literal | Edit state file to merge narrow patterns; the regex quality rules in the prompt help prevent this |\n| Spike keeps re-triggering | Pattern regex too broad (every log matches) | Tighten the regex in the state file; valid Python regex only |\n| Investigation never finishes | Agent stuck on a complex codebase | View conversation in OpenHands UI; clear `active_conversation` in state file to unblock next run |\n| Slack not receiving messages | `SLACK_BOT_TOKEN` missing `chat:write` or bot not in channel | Re-install app with correct scope; invite bot to channel |\n| State file write conflict | Script and agent writing concurrently (rare) | At worst one count update is lost; recovers automatically next run |" + }, { "name": "deno", "description": "If the project uses deno, use this skill. Use this skill to initialize and work with Deno projects, add/remove dependencies (JSR and npm), run tasks and scripts with appropriate permissions, and use built-in tooling (fmt/lint/test).",