Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changeset/collector-json-parse-noise.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'@hyperdx/otel-collector': patch
---

Stop the collector logging a `ParseJSON` syntax error for every log body that
contains braces but isn't JSON (e.g. Go struct or Python dict dumps). JSON
extraction now only matches objects that start with a quoted key, and remaining
parse failures in that statement group are silenced.
10 changes: 6 additions & 4 deletions docker/otel-collector/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,16 @@ processors:
transform:
log_statements:
- context: log
error_mode: ignore
# OTTL has no JSON validity check, so ParseJSON failures on non-JSON
# brace-delimited bodies are expected and would otherwise be logged.
error_mode: silent

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — error_mode: silent hides errors from the whole group, not just ParseJSON

Silent mode applies to every statement in this group, so failures in flatten(log.attributes) and merge_maps(log.attributes, log.body, "upsert") (lines 16-17) also stop being logged, even though nobody expects those to fail. Move the ParseJSON merge_maps statement (and the ExtractPatterns set it depends on) into its own context: log group with error_mode: silent, and keep flatten and the body-map merge in a group with error_mode: ignore so real failures there still show up in the logs.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — error_mode: silent hides errors from every statement in the group, not just ParseJSON

Put only the merge_maps(log.attributes, ParseJSON(log.cache["0"]), ...) statement in its own error_mode: silent group, and keep ignore for the ExtractPatterns, flatten(log.attributes) and merge_maps(log.attributes, log.body, ...) statements. As written, a real failure in those statements (for example a type error in the map-body merge, or a regex change that breaks ExtractPatterns) now leaves no log line. The comment and PR description only justify silencing the expected ParseJSON failures.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — error_mode: silent hides errors from every statement in the group, not just ParseJSON

The setting applies to the whole first log_statements group. That group also runs flatten(log.attributes) and merge_maps(log.attributes, log.body, "upsert") (lines 16-17), which run on map bodies and have nothing to do with JSON parsing. Any failure in those statements is now dropped without a trace. Move only the ParseJSON merge_maps statement (and the set(log.cache, ExtractPatterns...) it depends on) into its own group with error_mode: silent, and leave flatten and the body-map merge_maps in a group with error_mode: ignore.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

statements:
# JSON parsing: Extends log attributes with the fields from structured log body content, either as an OTEL map or
# as a string containing JSON content.
- set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{.*\\}))")) where
IsString(log.body)
- set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{\\s*\".*\\}))"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New extraction regex has no smoke-test fixture

Add records to smoke-tests/otel-collector/data/auto-parse/default/input.json and json-string/input.json (run by auto-parse-json.bats). Cover a Go struct dump {Name:x code:1} and a Python dict {'a': 1}, and assert they produce no attributes. Also cover [ctx] {thread=1} {"a":1}: the new \{\s*" anchor now extracts and parses this, but the old greedy match did not. Right now nothing pins the narrower regex, so a later change could quietly bring back the noise or drop real JSON.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

where IsString(log.body)
- merge_maps(log.attributes, ParseJSON(log.cache["0"]), "upsert")
where IsMap(log.cache)
where IsMap(log.cache) and IsString(log.cache["0"])
- flatten(log.attributes) where IsMap(log.cache)
- merge_maps(log.attributes, log.body, "upsert") where IsMap(log.body)
- context: log
Expand Down
19 changes: 19 additions & 0 deletions smoke-tests/otel-collector/auto-parse-json.bats
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,22 @@ load 'test_helpers/assertions.bash'
sleep 1
assert_test_data "data/auto-parse/default"
}

@test "brace-delimited non-JSON bodies are not merged and don't log parse errors" {
emit_otel_data "http://localhost:4318" "data/auto-parse/brace-not-json"
wait_for_rows 9000 "SELECT count() FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'brace-not-json'" 5
assert_test_data "data/auto-parse/brace-not-json"

run docker compose logs otel-collector
[ "$status" -eq 0 ]
# Prove we read the collector's output before asserting what's absent from it.
[[ "$output" == *"Everything is ready"* ]]
[[ "$output" != *"invalid character"* ]]
[[ "$output" != *"ParseJSON"* ]]
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
}

@test "JSON objects with whitespace after the opening brace are parsed" {
emit_otel_data "http://localhost:4318" "data/auto-parse/json-whitespace"
wait_for_rows 9000 "SELECT count() FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'json-whitespace'" 3
assert_test_data "data/auto-parse/json-whitespace"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
SELECT ResourceAttributes, LogAttributes FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'brace-not-json' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}"
"{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}"
"{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}"
"{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}"
"{'suite-id':'auto-parse','test-id':'brace-not-json'}","{}"
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
{
"resourceLogs": [
{
"resource": {
"attributes": [
{
"key": "suite-id",
"value": {
"stringValue": "auto-parse"
}
},
{
"key": "test-id",
"value": {
"stringValue": "brace-not-json"
}
}
]
},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"timeUnixNano": "1901999580000000000",
"body": {
"stringValue": "{Name:x Code:1}"
}
},
{
"timeUnixNano": "1901999580000000001",
"body": {
"stringValue": "{'a': 1}"
}
},
{
"timeUnixNano": "1901999580000000002",
"body": {
"stringValue": "{}"
}
},
{
"timeUnixNano": "1901999580000000003",
"body": {
"stringValue": "{ \"k\": unquoted }"
}
},
{
"timeUnixNano": "1901999580000000004",
"body": {
"stringValue": "{\"k\": \"v\""
}
}
]
}
]
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
SELECT ResourceAttributes, LogAttributes FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'auto-parse' AND ResourceAttributes['test-id'] = 'json-whitespace' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
"{'suite-id':'auto-parse','test-id':'json-whitespace'}","{'a':'1'}"
"{'suite-id':'auto-parse','test-id':'json-whitespace'}","{'b':'true'}"
"{'suite-id':'auto-parse','test-id':'json-whitespace'}","{'c':'d'}"
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{
"resourceLogs": [
{
"resource": {
"attributes": [
{
"key": "suite-id",
"value": {
"stringValue": "auto-parse"
}
},
{
"key": "test-id",
"value": {
"stringValue": "json-whitespace"
}
}
]
},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"timeUnixNano": "1901999580000000000",
"body": {
"stringValue": "{ \"a\": 1 }"
}
},
{
"timeUnixNano": "1901999580000000001",
"body": {
"stringValue": "{\t\"b\": true}"
}
},
{
"timeUnixNano": "1901999580000000002",
"body": {
"stringValue": "before {id:1} then { \"c\": \"d\" }"
}
}
]
}
]
}
]
}
Loading