feat(codecs): Add inject_metadata to VRL decoder + exec source support#25322
Closed
thomasqueirozb wants to merge 6 commits intomasterfrom
Closed
feat(codecs): Add inject_metadata to VRL decoder + exec source support#25322thomasqueirozb wants to merge 6 commits intomasterfrom
inject_metadata to VRL decoder + exec source support#25322thomasqueirozb wants to merge 6 commits intomasterfrom
Conversation
…secrets_template Allows sources to forward per-request secrets (e.g. authentication tokens) into the Deserializer pipeline so user-authored programs like VRL decoders can read them via get_secret!() during decoding. VrlDeserializer overrides parse_with_secrets to inject the secrets into the synthetic event before the VRL program executes, making them visible as %vector.secrets.* at runtime. All other Deserializer implementations use the default implementation which merges the template onto each emitted event after parsing, with the codec's own values taking priority.
Adds VrlDeserializerOptions.inject_metadata. When true, the source can call Decoder::with_metadata_template to pre-populate the synthetic event before VRL executes, making source context readable via % paths (e.g. %exec.host, %exec.command). VRL-produced values always win over injected values on collision. The exec source is the first consumer: it injects hostname and command into the decoder template at build time. Zero overhead when inject_metadata is false.
…ig::inject_metadata_enabled Move the inject_metadata check to DeserializerConfig::inject_metadata_enabled() so sources don't need VRL-specific knowledge. In handle_event, use try_insert for the vector-namespace metadata paths (host, command) when inject_metadata is enabled, so any value the VRL program wrote to %exec.host or %exec.command survives post-decode enrichment. No behavior change for Legacy namespace or when inject_metadata is false.
inject_metadata option for VRL decoder + exec source support
inject_metadata option for VRL decoder + exec source supportinject_metadata to VRL decoder + exec source support
9 tasks
Contributor
Author
|
If there is demand to support metadata on the VRL decoder for more sources this PR can be revived |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an
inject_metadataoption to thevrlcodec. When enabled, sources can callDecoder::with_metadata_templateto pre-populate the synthetic event with per-request context before the VRL program executes, making that context readable via%-prefixed paths during decoding.The
execsource is the first consumer: it injects%exec.hostand%exec.commandat build time, so VRL programs can read them withstring!(%exec.host)etc.Vector configuration
Basic — read injected metadata inside VRL decoder:
Collision test — VRL can read and overwrite the injected value within the program:
Output confirms
injected_host_after_override: "overridden-by-vrl"and thatmeta.host == "overridden-by-vrl"too.How did you test this PR?
cargo nextest run -E 'test(/vrl/)'(8 tests pass including two newwith_metadata_templatetests)Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Deserializer::parsetrait refactor)