Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ jobs:
uses: hyperi-io/hyperi-ci/.github/workflows/rust-ci.yml@main
with:
publish-target: both
# The filebeat test reads the pipeline and cases from this repo, beside the checkout.
sibling-checkouts: hyperi-io/dfe-transform-vrl
tag: ${{ inputs.tag || '' }}
from-head: ${{ inputs.from-head || '' }}
bump: ${{ inputs.bump || 'auto' }}
Expand Down
2 changes: 1 addition & 1 deletion .gitleaks.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,6 @@ description = "Project-specific false positive allowlist"
paths = [
'''(^|/)ci/''',
'''(^|/)ai/''',
# Commits before the history rewrite still hold the deleted file, and gitleaks scans them.
# The Meraki event subtype '8021x_eap_failure' matches a key rule.
'''pipelines/filebeat/filebeat\.yaml''',
]
4 changes: 3 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ A Rust supervisor that runs Vector as a child process so a Vector pipeline behav
| `src/health.rs`, `src/metrics.rs`, `src/metrics/scrape.rs` | Readiness publishing, and the scrape that merges `vector_*` into scalo's registry |
| `src/deployment.rs`, `src/vector-layer.dockerfile` | `contract()`, the source the Dockerfile and the released chart are generated from, and the two-binary image override spliced onto scalo's generated Dockerfile |
| `templates/bus.yaml`, `templates/direct.yaml` | Whole runnable Vector topologies, one per transport, commented per field |
| `pipelines/filebeat/` | The bundled filebeat pipeline. The image does not carry it |
| `docs/` | `architecture.md` for why the service is built this way and the invariants, `internals.md` for field-by-field depth, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` |
| `tests/` | `integration`, `e2e`, `smoke`, and `TESTING.md` for how the broker and Vector binary are resolved |
| `scripts/fetch-vector.sh`, `scripts/pgo-workload.sh` | Downloads the pinned Vector for tests, and drives the PGO/BOLT workload |
Expand All @@ -170,7 +171,7 @@ What green does NOT mean:

- The default run leaves out the `#[ignore]` cases in `tests/integration/vector_validate.rs` and `tests/e2e/kafka.rs`. The `vector_validate` ones run Vector's own validator over each buffer, SASL and TLS variant of an assembled config, so until you pass `--run-ignored all` those variants are unchecked.
- Locally, a missing broker or Vector binary SKIPS rather than fails. In CI that absence is an assertion failure for the cases that run by default, so CI is stricter than a green laptop run.
- `e2e::app_kafka` runs by default: the shipped binary over a shared broker with an authored transform file. Without Docker it skips locally and fails under CI.
- `e2e::filebeat_kafka` runs by default and needs a dfe-transform-vrl checkout, for its cases and the bundled pipeline: beside this repo, or named by `DFE_TRANSFORM_VRL_DIR`. Without one it skips locally and fails under CI, which checks the repo out beside this one.
- A docs-only push runs no CI at all. `paths-ignore` for `docs/**` and `**.md` is on the `push` trigger but not on `pull_request`, so the PR is the only place a docs change is checked. A silent push is not a pass.

### What tends to bite
Expand Down Expand Up @@ -208,3 +209,4 @@ Outbound -- what depends on this repo:
Two more relationships that are real but are NOT declared edges in the suite graph, so no gate enforces them:

- dfe-infra `apps.yaml` is the SSoT for how this app is deployed -- multiplicity, scaling, transports, endpoints and which compiler derives its routing. dfe-engine reads that manifest, so adding or changing the app is a manifest edit, never an engine release.
- dfe-transform-vrl holds the filebeat cases (`tests/fixtures/filebeat/cases.yaml`) and the bundled pipeline that `e2e::filebeat_kafka` grades against. The dependency is a test-time checkout only, which CI makes and a laptop may not.
36 changes: 36 additions & 0 deletions pipelines/filebeat/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
<!--
Project: dfe-transform-vector
File: pipelines/filebeat/README.md
Purpose: How the bundled filebeat pipeline is used and where it comes from
Language: Markdown

License: BUSL-1.1
Copyright: (c) 2026 HYPERI PTY LIMITED
-->

# Bundled pipeline: filebeat-compat (INTERIM)

A pre-canned transform file that keeps DFE 2.1 filebeat feeds working on dfe-transform-vector: the same VRL dfe-transform-vrl runs, wrapped in a Vector `remap` and carrying the enrichment table it needs. Point `transforms.dir` at this directory, mount the table, and the pipeline runs -- there is nothing to author. The modules, the routing rules, the departures from DFE 2.1 and the known limitations are documented once, in dfe-transform-vrl `pipelines/filebeat/README.md`, and apply here unchanged.

> **INTERIM.** Elastic compatibility is being replaced by
> `dfe-transform-elastic` (Rust-native). New integrations
> should NOT build on this pipeline.

## Using it

```yaml
# config.yaml
dfe_source: "filebeat" # derives filebeat_land -> filebeat_load
transforms:
dir: "/etc/dfe-transform-vector/transforms"
```

Mount `filebeat.yaml` into that directory and `timezones.csv` at the path the pipeline's `enrichment_tables` entry names (`/etc/dfe-transform-vector/data/timezones.csv`), or edit that one path to wherever the table lives inside `transforms.data_dir`. Both files are here.

The wrapper writes each transform file into Vector's config dir with its wired `inputs` added, and a transform file may declare a top-level `enrichment_tables` key, so Vector merges the declaration and the VRL's timezone lookups resolve. `vector validate` runs before Vector starts, so a pipeline that cannot compile stops the app rather than passing nothing on.

Events carry the DFE 2.1 Kafka fields (`{message, tags, timestamp}`), and the optional `._conf.tz_offset` / `._conf.tz_map` fields drive the ios and meraki timezone handling.

## Provenance

`filebeat.yaml` embeds dfe-transform-vrl `pipelines/filebeat/filebeat.vrl` @ `e0b7087`, the same program bar two lines of its header comment, and `timezones.csv` is that repo's table copied. Refresh by copying both again rather than editing here -- the two transform apps grade against the same cases (`tests/e2e/filebeat_kafka.rs` in each), and that only means something while the program is the same one.
Loading
Loading