Skip to content

feat(codecs): Add inject_metadata to VRL decoder + exec source support#25322

Closed
thomasqueirozb wants to merge 6 commits intomasterfrom
decoder-metadata-template
Closed

feat(codecs): Add inject_metadata to VRL decoder + exec source support#25322
thomasqueirozb wants to merge 6 commits intomasterfrom
decoder-metadata-template

Conversation

@thomasqueirozb
Copy link
Copy Markdown
Contributor

@thomasqueirozb thomasqueirozb commented Apr 28, 2026

Summary

Adds an inject_metadata option to the vrl codec. When enabled, sources can call Decoder::with_metadata_template to pre-populate the synthetic event with per-request context before the VRL program executes, making that context readable via %-prefixed paths during decoding.

The exec source is the first consumer: it injects %exec.host and %exec.command at build time, so VRL programs can read them with string!(%exec.host) etc.

Vector configuration

Basic — read injected metadata inside VRL decoder:

sources:
  exec_source:
    type: exec
    mode: scheduled
    scheduled:
      exec_interval_secs: 5
    command:
      - echo
      - '{"level":"info","msg":"hello from exec"}'
    framing:
      method: newline_delimited
    decoding:
      codec: vrl
      vrl:
        inject_metadata: true
        source: |
          . = parse_json!(string!(.message))
          .decoder_saw_host = get!(%exec.host, [])
          .decoder_saw_command = get!(%exec.command, [])
sinks:
  console:
    type: console
    inputs: [exec_source]
    encoding:
      codec: json

Collision test — VRL can read and overwrite the injected value within the program:

sources:
  exec_source:
    type: exec
    mode: scheduled
    scheduled:
      exec_interval_secs: 5
    command: [echo, "ignored payload"]
    framing:
      method: newline_delimited
    decoding:
      codec: vrl
      vrl:
        inject_metadata: true
        source: |
          .injected_host_before_override = string!(%exec.host)
          %exec.host = "overridden-by-vrl"
          .injected_host_after_override = string!(%exec.host)
          . = {
            "injected_host_before_override": .injected_host_before_override,
            "injected_host_after_override": .injected_host_after_override,
          }

transforms:
  inject_meta:
    inputs: [exec_source]
    type: remap
    source: |
      .meta = %

sinks:
  console:
    type: console
    inputs: [inject_meta]
    encoding:
      codec: json

Output confirms injected_host_after_override: "overridden-by-vrl" and that meta.host == "overridden-by-vrl" too.

How did you test this PR?

  • Unit tests: cargo nextest run -E 'test(/vrl/)' (8 tests pass including two new with_metadata_template tests)
  • Live: ran both configs above against the built binary

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

…secrets_template

Allows sources to forward per-request secrets (e.g. authentication tokens) into
the Deserializer pipeline so user-authored programs like VRL decoders can read
them via get_secret!() during decoding.

VrlDeserializer overrides parse_with_secrets to inject the secrets into the
synthetic event before the VRL program executes, making them visible as
%vector.secrets.* at runtime. All other Deserializer implementations use the
default implementation which merges the template onto each emitted event after
parsing, with the codec's own values taking priority.
@thomasqueirozb thomasqueirozb added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Apr 28, 2026
@github-actions github-actions Bot added the domain: core Anything related to core crates i.e. vector-core, core-common, etc label Apr 28, 2026
Adds VrlDeserializerOptions.inject_metadata. When true, the source can call
Decoder::with_metadata_template to pre-populate the synthetic event before VRL
executes, making source context readable via % paths (e.g. %exec.host,
%exec.command). VRL-produced values always win over injected values on collision.

The exec source is the first consumer: it injects hostname and command into the
decoder template at build time. Zero overhead when inject_metadata is false.
@github-actions github-actions Bot added work in progress domain: sources Anything related to the Vector's sources domain: external docs Anything related to Vector's external, public documentation labels Apr 29, 2026
@thomasqueirozb thomasqueirozb changed the title feat(codecs): add Deserializer::parse_with_secrets and Decoder::with_secrets_template feat(codecs): inject_metadata option for the VRL decoder Apr 29, 2026
…ig::inject_metadata_enabled

Move the inject_metadata check to DeserializerConfig::inject_metadata_enabled() so
sources don't need VRL-specific knowledge. In handle_event, use try_insert for the
vector-namespace metadata paths (host, command) when inject_metadata is enabled, so
any value the VRL program wrote to %exec.host or %exec.command survives
post-decode enrichment. No behavior change for Legacy namespace or when
inject_metadata is false.
@thomasqueirozb thomasqueirozb changed the title feat(codecs): inject_metadata option for the VRL decoder feat(codecs): inject_metadata option for VRL decoder; wired up in exec source Apr 29, 2026
@thomasqueirozb thomasqueirozb changed the title feat(codecs): inject_metadata option for VRL decoder; wired up in exec source feat(codecs): Add inject_metadata option for VRL decoder + exec source support Apr 29, 2026
@thomasqueirozb thomasqueirozb changed the title feat(codecs): Add inject_metadata option for VRL decoder + exec source support feat(codecs): Add inject_metadata to VRL decoder + exec source support Apr 29, 2026
@thomasqueirozb
Copy link
Copy Markdown
Contributor Author

If there is demand to support metadata on the VRL decoder for more sources this PR can be revived

@github-actions github-actions Bot locked and limited conversation to collaborators Apr 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: external docs Anything related to Vector's external, public documentation domain: sources Anything related to the Vector's sources no-changelog Changes in this PR do not need user-facing explanations in the release changelog work in progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant