security: GCP Log Monitoring Completely Broken — 3 Consecutive Hourly Scans Failed (Monitoring Blackout Since 2026-04-10T00:00Z)

[vks-archastro]

🚨 Security Monitoring Blackout

The automated GCP log monitoring agent has failed 3 consecutive times (scheduled runs at 00:00, 01:00, and 02:00 UTC on 2026-04-10). No security telemetry has been collected for the past 2+ hours. Auth failures, privilege escalations, and suspicious access events in this window are completely undetected.

---

Failure Details

Run	Scheduled At	Status	Error
1	2026-04-10T00:00Z	❌ FAILED	{:invalid_arguments, "string.replace", "expected different arguments, got: null, string, string"}
2	2026-04-10T01:00Z	❌ FAILED	Same error — persistent
3	2026-04-10T02:00Z	❌ FAILED	Same error — confirmed persistent

Root Cause: The query_gcloud_logs tool receives a null value for project_id at runtime. The GCP project binding is not configured in the agent's environment. All three query types (auth failures, privilege escalations, suspicious access) fail with the same error.

---

Queries That Are Failing

1. Authentication Failures (gRPC codes 7 & 16)

protoPayload.status.code = 7 OR protoPayload.status.code = 16
log_id("cloudaudit.googleapis.com/activity")

Detects: PERMISSION_DENIED (7) and UNAUTHENTICATED (16) — brute force, stolen credentials, token abuse.

2. Privilege Escalation

protoPayload.methodName = "SetIamPolicy"
  OR "google.iam.admin.v1.CreateServiceAccountKey"
  OR "google.iam.admin.v1.SignJwt"
log_id("cloudaudit.googleapis.com/activity")

Detects: IAM policy changes, new SA keys, JWT signing abuse.

3. Suspicious Access Patterns

protoPayload.authorizationInfo.granted = false OR severity = "ERROR"
log_id("cloudaudit.googleapis.com/data_access")

Detects: unauthorized data access attempts, API errors indicative of probing.

---

Impact

• Window unmonitored: 2026-04-10 00:00 UTC → ongoing (2+ hours and growing every scheduled run)
• Threat signals missed: Auth brute force, credential stuffing, insider privilege abuse, lateral movement
• Compliance risk: Continuous monitoring requirements are not being met during this blackout

---

Required Actions

Immediate (Human — Do Now)

1. Manually review GCP Cloud Audit Logs for the 2026-04-10 00:00–02:00 UTC window using the queries above in the Cloud Console or gcloud CLI
2. Check for: repeated PERMISSION_DENIED/UNAUTHENTICATED from same IPs/principals, SetIamPolicy calls, CreateServiceAccountKey events, denied data access
3. Confirm no active incident is in progress before treating this as a configuration issue

Fix (Engineering)

1. Configure GCP project ID in the agent's runtime environment — the GCP_PROJECT_ID env var (or equivalent) resolves to null
2. Verify query_gcloud_logs tool binding — ensure the tool can resolve project context at invocation time
3. Add a health check — the monitoring agent should alert immediately if project_id is null rather than silently failing
4. Test the fix by running a manual scan before the next scheduled window

Process

• Add a dead-man's-switch alert: if 2+ consecutive monitoring runs fail, page on-call immediately
• Consider alerting directly from GCP (log-based alerts) as a backup to this agent

---

Triage Reasoning

Escalating instead of self-fixing because:

• The project_id binding is an infrastructure/configuration problem outside the agent's scope
• The agent cannot configure its own environment variables
• 2+ hours of monitoring blackout is a material security gap requiring human acknowledgment
• A human must manually cover the unmonitored window — there is no automated fallback

---

References

• Prior monitoring failure issues recorded in agent memory (runs at 00:00 and 01:00 UTC)
• GCP gRPC status code 7 = PERMISSION_DENIED, code 16 = UNAUTHENTICATED
• Cloud Audit Logs documentation
• IAM Audit Logging