diff --git a/docs/detectors.md b/docs/detectors.md index 5dad456..3123155 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -88,43 +88,63 @@ List of detectors: * [New Value](detectors/new_value.md): Detect new values in the variables in the logs. * [Combo Detector](detectors/combo.md): Detect new combination of variables in the logs. +## Configuration -## Auto-configuration (optional) - -Detectors can optionally support **auto-configuration** — a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually. - -### Enabling auto-configuration - -Auto-configuration is controlled by the `auto_config` flag in the pipeline config (e.g. `config/pipeline_config_default.yaml`): +When `auto_config` is set to `False`, the detector expects an explicit `events` block that specifies exactly which variables to monitor: ```yaml detectors: NewValueDetector: method_type: new_value_detector - auto_config: True # enable auto-configuration - params: {} - # no "events" block needed — it will be generated automatically + auto_config: False + params: {} # global parameters + events: # event-specific configuration + 1: # event_id + instance1: # name of instance (arbitrary) + params: {} # additional params + variables: + - pos: 0 # location of an unnamed variable from the log message + name: var1 # name of variable (arbitrary) + header_variables: + - pos: level # location of a named variable (defined in log_format of parser) + global: # define global instance for new_value_detector similar to "events" + global_instance1: # define instance name + header_variables: # same logic as header_variables in "events" + - pos: Status ``` -When `auto_config` is set to `False`, the detector expects an explicit `events` block that specifies exactly which variables to monitor: + +### Configuration semantics (preliminary) + +**`events` key** — The integer key is the `EventID` (or `event_id`) to monitor (see the MatcherParser docs for how EventID is assigned). + +**`variables[].pos`** — The 0-indexed position of the `<*>` wildcard in the matched template, counting from left to right starting at 0. For example, given: + +```text +pid=<*> uid=<*> auid=<*> ses=<*> msg='op=<*> acct=<*> exe=<*> hostname=<*> addr=<*> terminal=<*> res=<*>' +``` + +`pos: 0` captures `pid=`, `pos: 6` captures `exe=`, etc. + +**`header_variables[].pos`** — A named field from the log format string (e.g., `Type`, `Time`, `Content`) rather than a wildcard position. + + +### Auto-configuration (optional) + +Detectors can optionally support **auto-configuration** — a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually. + +Auto-configuration is controlled by the `auto_config` flag in the pipeline config (e.g. `config/pipeline_config_default.yaml`): ```yaml detectors: NewValueDetector: method_type: new_value_detector - auto_config: False + auto_config: True # enable auto-configuration params: {} - events: - 1: - instance1: - params: {} - variables: - - pos: 0 - name: var1 - header_variables: - - pos: level + # no "events" block needed — it will be generated automatically ``` + ### How it works When auto-configuration is enabled, the detector goes through two extra phases before training: diff --git a/docs/parsers/template_matcher.md b/docs/parsers/template_matcher.md index dc42bf7..9b8a64c 100644 --- a/docs/parsers/template_matcher.md +++ b/docs/parsers/template_matcher.md @@ -14,9 +14,23 @@ The template matcher is a lightweight, fast parser intended for logs that follow - Preprocesses logs and templates (remove spaces, punctuation, lowercase) based on config. - Finds the first template that matches and extracts all wildcard parameters in order. - Populates ParserSchema fields: `EventID`, `template`, `variables`, `logID`, and related fields. +- **`EventID` is the 0-indexed line number of the matched template** in the template file (first line → `EventID: 0`, second line → `EventID: 1`, etc.). This parser is deterministic and designed for high-throughput use when templates are known in advance. +## EventID assignment (preliminary) + +The `EventID` (or `event_id`) field in the output `ParserSchema` identifies which template was matched. It equals the **0-indexed line number** of the matching template in the template file: + +| Line in template file | EventID | +|-----------------------|---------| +| 1st line | 0 | +| 2nd line | 1 | +| 3rd line | 2 | +| ... | ... | + +This `EventID` is the integer key used in detector configurations (e.g., `NewValueDetector`) to scope detection rules to logs of a particular template type. + ## Template format - Templates are plain text lines in a template file.