From 46c1b146ddf738104f96111754cd583bc3427ae5 Mon Sep 17 00:00:00 2001 From: viktorbeck98 Date: Tue, 17 Mar 2026 21:48:17 +0100 Subject: [PATCH 1/2] update config docs --- docs/detectors.md | 50 +++++++++++++++++++++----------- docs/parsers/template_matcher.md | 14 +++++++++ 2 files changed, 47 insertions(+), 17 deletions(-) diff --git a/docs/detectors.md b/docs/detectors.md index 5dad456..f0c1712 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -88,23 +88,7 @@ List of detectors: * [New Value](detectors/new_value.md): Detect new values in the variables in the logs. * [Combo Detector](detectors/combo.md): Detect new combination of variables in the logs. - -## Auto-configuration (optional) - -Detectors can optionally support **auto-configuration** — a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually. - -### Enabling auto-configuration - -Auto-configuration is controlled by the `auto_config` flag in the pipeline config (e.g. `config/pipeline_config_default.yaml`): - -```yaml -detectors: - NewValueDetector: - method_type: new_value_detector - auto_config: True # enable auto-configuration - params: {} - # no "events" block needed — it will be generated automatically -``` +## Configuration When `auto_config` is set to `False`, the detector expects an explicit `events` block that specifies exactly which variables to monitor: @@ -125,6 +109,38 @@ detectors: - pos: level ``` + +### Configuration semantics (preliminary) + +**`events` key** — The integer key is the `EventID` (or `event_id`) to monitor (see the MatcherParser docs for how EventID is assigned). + +**`variables[].pos`** — The 0-indexed position of the `<*>` wildcard in the matched template, counting from left to right starting at 0. For example, given: + +```text +pid=<*> uid=<*> auid=<*> ses=<*> msg='op=<*> acct=<*> exe=<*> hostname=<*> addr=<*> terminal=<*> res=<*>' +``` + +`pos: 0` captures `pid=`, `pos: 6` captures `exe=`, etc. + +**`header_variables[].pos`** — A named field from the log format string (e.g., `Type`, `Time`, `Content`) rather than a wildcard position. + + +### Auto-configuration (optional) + +Detectors can optionally support **auto-configuration** — a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually. + +Auto-configuration is controlled by the `auto_config` flag in the pipeline config (e.g. `config/pipeline_config_default.yaml`): + +```yaml +detectors: + NewValueDetector: + method_type: new_value_detector + auto_config: True # enable auto-configuration + params: {} + # no "events" block needed — it will be generated automatically +``` + + ### How it works When auto-configuration is enabled, the detector goes through two extra phases before training: diff --git a/docs/parsers/template_matcher.md b/docs/parsers/template_matcher.md index dc42bf7..9b8a64c 100644 --- a/docs/parsers/template_matcher.md +++ b/docs/parsers/template_matcher.md @@ -14,9 +14,23 @@ The template matcher is a lightweight, fast parser intended for logs that follow - Preprocesses logs and templates (remove spaces, punctuation, lowercase) based on config. - Finds the first template that matches and extracts all wildcard parameters in order. - Populates ParserSchema fields: `EventID`, `template`, `variables`, `logID`, and related fields. +- **`EventID` is the 0-indexed line number of the matched template** in the template file (first line → `EventID: 0`, second line → `EventID: 1`, etc.). This parser is deterministic and designed for high-throughput use when templates are known in advance. +## EventID assignment (preliminary) + +The `EventID` (or `event_id`) field in the output `ParserSchema` identifies which template was matched. It equals the **0-indexed line number** of the matching template in the template file: + +| Line in template file | EventID | +|-----------------------|---------| +| 1st line | 0 | +| 2nd line | 1 | +| 3rd line | 2 | +| ... | ... | + +This `EventID` is the integer key used in detector configurations (e.g., `NewValueDetector`) to scope detection rules to logs of a particular template type. + ## Template format - Templates are plain text lines in a template file. From 5f53b97a74791a8f769eb50e9b90045e93347896 Mon Sep 17 00:00:00 2001 From: viktorbeck98 Date: Tue, 17 Mar 2026 21:58:50 +0100 Subject: [PATCH 2/2] add inline comments for config --- docs/detectors.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/detectors.md b/docs/detectors.md index f0c1712..3123155 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -97,16 +97,20 @@ detectors: NewValueDetector: method_type: new_value_detector auto_config: False - params: {} - events: - 1: - instance1: - params: {} + params: {} # global parameters + events: # event-specific configuration + 1: # event_id + instance1: # name of instance (arbitrary) + params: {} # additional params variables: - - pos: 0 - name: var1 + - pos: 0 # location of an unnamed variable from the log message + name: var1 # name of variable (arbitrary) header_variables: - - pos: level + - pos: level # location of a named variable (defined in log_format of parser) + global: # define global instance for new_value_detector similar to "events" + global_instance1: # define instance name + header_variables: # same logic as header_variables in "events" + - pos: Status ```