Skip to content

fix(otel-collector): stop logging ParseJSON errors for non-JSON log bodies - #3336

Merged
kodiakhq[bot] merged 4 commits into
mainfrom
warren/fix-collector-log-statement-processor
Oct 14, 2026
Merged

kodiakhq[bot] merged 4 commits into
mainfrom
warren/fix-collector-log-statement-processor

Conversation

@wrn14897

@wrn14897 wrn14897 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Why

The collector's log transform extracts any {...} substring from a string body and passes it to ParseJSON. Bodies that contain braces but aren't JSON (Go struct dumps, Python dicts, {"k": unquoted}) fail to parse, and error_mode: ignore logs every failure:

error getting value in ottl.StandardPMapGetter[...ottllog.TransformContext]: invalid character 'c' looking for beginning of value
exception.type: *errors.SyntaxError
statement: merge_maps(log.attributes, ParseJSON(log.cache["0"]), "upsert") where IsMap(log.cache) and IsString(log.cache["0"])

OTTL has no JSON validity check, so these failures can't be guarded away entirely.

What

In the JSON-parsing statement group of docker/otel-collector/config.yaml:

  • The extraction regex now requires { followed by a quoted key (\{\s*".*\}), so struct/dict dumps never reach ParseJSON.
  • Add IsString(log.cache["0"]) to the merge_maps guard (matches hyperdx-ee).
  • Set that group's error_mode to silent; remaining parse failures are expected noise. Other groups keep ignore.

Testing

Ran otel/opentelemetry-collector-contrib:0.155.0 with this transform and an OTLP→debug pipeline:

  • Old config: {"k": cbad} logs invalid character 'c'.
  • New config: {Name:x code:1}, {'a': 1}, {"k": cbad} produce no errors.
  • prefix {"level":"warn","foo":"bar"} still merges level/foo into attributes and sets severity warn.

@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 14, 2026 9:04pm UTC
hyperdx-storybook Ready Ready Preview Oct 14, 2026 9:04pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 14688f0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/otel-collector Patch
@hyperdx/api Patch
@hyperdx/app Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Additional context: touches background tasks or the delivery pipeline lightly (10 lines, under the 30-line bar for Tier 4)

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 8
  • Production lines changed: 145
  • Branch: warren/fix-collector-log-statement-processor
  • Author: wrn14897

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 124 in total (1.71%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

error_mode: ignore
# OTTL has no JSON validity check, so ParseJSON failures on non-JSON
# brace-delimited bodies are expected and would otherwise be logged.
error_mode: silent

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — error_mode: silent hides errors from the whole group, not just ParseJSON

Silent mode applies to every statement in this group, so failures in flatten(log.attributes) and merge_maps(log.attributes, log.body, "upsert") (lines 16-17) also stop being logged, even though nobody expects those to fail. Move the ParseJSON merge_maps statement (and the ExtractPatterns set it depends on) into its own context: log group with error_mode: silent, and keep flatten and the body-map merge in a group with error_mode: ignore so real failures there still show up in the logs.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

# as a string containing JSON content.
- set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{.*\\}))")) where
IsString(log.body)
- set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{\\s*\".*\\}))"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New extraction regex has no smoke-test fixture

Add records to smoke-tests/otel-collector/data/auto-parse/default/input.json and json-string/input.json (run by auto-parse-json.bats). Cover a Go struct dump {Name:x code:1} and a Python dict {'a': 1}, and assert they produce no attributes. Also cover [ctx] {thread=1} {"a":1}: the new \{\s*" anchor now extracts and parses this, but the old greedy match did not. Right now nothing pins the narrower regex, so a later change could quietly bring back the noise or drop real JSON.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

1 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues remain.

Summary

The collector now skips obvious non-JSON brace bodies and silences remaining JSON parse errors.

  • Adds a string check before ParseJSON.
  • Adds smoke tests for malformed bodies and JSON with whitespace.
  • Addresses the earlier unnumbered finding: failed log reads now fail the test.

Reviews (4) · Last reviewed commit: "Merge branch 'main' into warren/fix-coll..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: otel-collector OTTL log-transform fix (error_mode, JSON-extraction regex, merge guard) plus bats smoke tests and fixtures. Mode: report-only. Base: 0b52907.

✅ No critical issues found. The regex tightening, the added IsString(log.cache["0"]) guard, and the error_mode switch were traced against every new and existing fixture: valid JSON objects (which always open with { + optional whitespace + a quoted key) still match, non-JSON brace bodies no longer reach ParseJSON, and the unchanged json-string/default/otel-map snapshots remain correct. The prior greptile $status comment is addressed (the author added [ "$status" -eq 0 ] plus a positive "Everything is ready" assertion).

🟡 P2 — recommended

  • docker/otel-collector/config.yaml:8 — error_mode: silent is group-scoped, so it suppresses error logs for every statement in the JSON group (ExtractPatterns, ParseJSON, flatten, and the final merge_maps(log.attributes, log.body)), not only the intended non-JSON ParseJSON noise; a genuine future misconfiguration in those other statements would become invisible to operators while data still flows.
    • Fix: Either extend the inline comment to state the group-wide suppression is deliberate, or isolate the ParseJSON statement into its own context: log group so the remaining statements keep error_mode: ignore.
    • maintainability, reliability
🔵 P3 nitpicks (2)
  • smoke-tests/otel-collector/auto-parse-json.bats:34 — the != *"ParseJSON"* / != *"invalid character"* absence assertions verify the error_mode: silent change but not the regex tightening, since group-level silent would swallow those logs even under the old loose pattern; the behavioral proof actually comes from the expected.snap showing empty LogAttributes.
    • Fix: Note in a comment that these absence checks guard the error_mode change specifically, relying on the snapshot assertion for the regex behavior.
  • smoke-tests/otel-collector/auto-parse-json.bats:29 — docker compose logs otel-collector returns cumulative logs from every prior test against the shared collector, so a future non-silenced test that legitimately emits ParseJSON/invalid character would break this unrelated assertion.
    • Fix: Accept the coupling with a comment, or scope the log check to lines correlated with this test's emitted bodies.

Reviewers (8): correctness, testing, maintainability, reliability, project-standards, previous-comments, agent-native, learnings.

Testing gaps:

  • No fixture covers greedy over-match where a valid JSON object is followed on the same line by trailing non-JSON ending in } (e.g. {"a":1} foo }) or by a second valid object ({"a":1} {"b":2}); the greedy .*\} captures the whole span, ParseJSON fails, and the attributes are silently dropped. Pre-existing (the old \{.*\} was equally greedy), now quieter under silent.
  • Multi-line brace-delimited bodies are untested — RE2 . does not cross newlines, so a JSON object spanning newlines is not extracted (pre-existing behavior, undocumented by test).

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 416 passed • 1 skipped • 1294s

Status Count
✅ Passed 416
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

error_mode: ignore
# OTTL has no JSON validity check, so ParseJSON failures on non-JSON
# brace-delimited bodies are expected and would otherwise be logged.
error_mode: silent

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — error_mode: silent hides errors from every statement in the group, not just ParseJSON

Put only the merge_maps(log.attributes, ParseJSON(log.cache["0"]), ...) statement in its own error_mode: silent group, and keep ignore for the ExtractPatterns, flatten(log.attributes) and merge_maps(log.attributes, log.body, ...) statements. As written, a real failure in those statements (for example a type error in the map-body merge, or a regex change that breaks ExtractPatterns) now leaves no log line. The comment and PR description only justify silencing the expected ParseJSON failures.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

error_mode: ignore
# OTTL has no JSON validity check, so ParseJSON failures on non-JSON
# brace-delimited bodies are expected and would otherwise be logged.
error_mode: silent

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — error_mode: silent hides errors from every statement in the group, not just ParseJSON

The setting applies to the whole first log_statements group. That group also runs flatten(log.attributes) and merge_maps(log.attributes, log.body, "upsert") (lines 16-17), which run on map bodies and have nothing to do with JSON parsing. Any failure in those statements is now dropped without a trace. Move only the ParseJSON merge_maps statement (and the set(log.cache, ExtractPatterns...) it depends on) into its own group with error_mode: silent, and leave flatten and the body-map merge_maps in a group with error_mode: ignore.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@kodiakhq
kodiakhq Bot merged commit ab563d2 into main Oct 14, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the warren/fix-collector-log-statement-processor branch October 14, 2026 21:10

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 14688f01 Deployed Oct 14, 2026 by vercel[bot]
Preview – hyperdx-storybook — 14688f01 Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants