Repository navigation
fix(otel-collector): stop logging ParseJSON errors for non-JSON log bodies #3336
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
4563f45
a997300
47610d8
14688f0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| --- | ||
| '@hyperdx/otel-collector': patch | ||
| --- | ||
|
|
||
| Stop the collector logging a `ParseJSON` syntax error for every log body that | ||
| contains braces but isn't JSON (e.g. Go struct or Python dict dumps). JSON | ||
| extraction now only matches objects that start with a quoted key, and remaining | ||
| parse failures in that statement group are silenced. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,14 +3,16 @@ processors: | |
| transform: | ||
| log_statements: | ||
| - context: log | ||
| error_mode: ignore | ||
| # OTTL has no JSON validity check, so ParseJSON failures on non-JSON | ||
| # brace-delimited bodies are expected and would otherwise be logged. | ||
| error_mode: silent | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — error_mode: silent hides errors from every statement in the group, not just ParseJSON Put only the Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — The setting applies to the whole first Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| statements: | ||
| # JSON parsing: Extends log attributes with the fields from structured log body content, either as an OTEL map or | ||
| # as a string containing JSON content. | ||
| - set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{.*\\}))")) where | ||
| IsString(log.body) | ||
| - set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{\\s*\".*\\}))")) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — New extraction regex has no smoke-test fixture Add records to Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| where IsString(log.body) | ||
| - merge_maps(log.attributes, ParseJSON(log.cache["0"]), "upsert") | ||
| where IsMap(log.cache) | ||
| where IsMap(log.cache) and IsString(log.cache["0"]) | ||
| - flatten(log.attributes) where IsMap(log.cache) | ||
| - merge_maps(log.attributes, log.body, "upsert") where IsMap(log.body) | ||
| - context: log | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| SELECT ResourceAttributes, LogAttributes FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'brace-not-json' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| "{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}" | ||
| "{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}" | ||
| "{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}" | ||
| "{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}" | ||
| "{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| { | ||
| "resourceLogs": [ | ||
| { | ||
| "resource": { | ||
| "attributes": [ | ||
| { | ||
| "key": "suite-id", | ||
| "value": { | ||
| "stringValue": "auto-parse" | ||
| } | ||
| }, | ||
| { | ||
| "key": "test-id", | ||
| "value": { | ||
| "stringValue": "brace-not-json" | ||
| } | ||
| } | ||
| ] | ||
| }, | ||
| "scopeLogs": [ | ||
| { | ||
| "scope": {}, | ||
| "logRecords": [ | ||
| { | ||
| "timeUnixNano": "1901999580000000000", | ||
| "body": { | ||
| "stringValue": "{Name:x Code:1}" | ||
| } | ||
| }, | ||
| { | ||
| "timeUnixNano": "1901999580000000001", | ||
| "body": { | ||
| "stringValue": "{'a': 1}" | ||
| } | ||
| }, | ||
| { | ||
| "timeUnixNano": "1901999580000000002", | ||
| "body": { | ||
| "stringValue": "{}" | ||
| } | ||
| }, | ||
| { | ||
| "timeUnixNano": "1901999580000000003", | ||
| "body": { | ||
| "stringValue": "{ \"k\": unquoted }" | ||
| } | ||
| }, | ||
| { | ||
| "timeUnixNano": "1901999580000000004", | ||
| "body": { | ||
| "stringValue": "{\"k\": \"v\"" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| SELECT ResourceAttributes, LogAttributes FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'json-whitespace' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| "{'suite-id':'auto-parse','test-id':'json-whitespace'}","{'a':'1'}" | ||
| "{'suite-id':'auto-parse','test-id':'json-whitespace'}","{'b':'true'}" | ||
| "{'suite-id':'auto-parse','test-id':'json-whitespace'}","{'c':'d'}" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| { | ||
| "resourceLogs": [ | ||
| { | ||
| "resource": { | ||
| "attributes": [ | ||
| { | ||
| "key": "suite-id", | ||
| "value": { | ||
| "stringValue": "auto-parse" | ||
| } | ||
| }, | ||
| { | ||
| "key": "test-id", | ||
| "value": { | ||
| "stringValue": "json-whitespace" | ||
| } | ||
| } | ||
| ] | ||
| }, | ||
| "scopeLogs": [ | ||
| { | ||
| "scope": {}, | ||
| "logRecords": [ | ||
| { | ||
| "timeUnixNano": "1901999580000000000", | ||
| "body": { | ||
| "stringValue": "{ \"a\": 1 }" | ||
| } | ||
| }, | ||
| { | ||
| "timeUnixNano": "1901999580000000001", | ||
| "body": { | ||
| "stringValue": "{\t\"b\": true}" | ||
| } | ||
| }, | ||
| { | ||
| "timeUnixNano": "1901999580000000002", | ||
| "body": { | ||
| "stringValue": "before {id:1} then { \"c\": \"d\" }" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 minor —
error_mode: silenthides errors from the whole group, not just ParseJSONSilent mode applies to every statement in this group, so failures in
flatten(log.attributes)andmerge_maps(log.attributes, log.body, "upsert")(lines 16-17) also stop being logged, even though nobody expects those to fail. Move theParseJSONmerge_mapsstatement (and theExtractPatternsset it depends on) into its owncontext: loggroup witherror_mode: silent, and keepflattenand the body-map merge in a group witherror_mode: ignoreso real failures there still show up in the logs.Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond