Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 41 additions & 21 deletions docs/detectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,43 +88,63 @@ List of detectors:
* [New Value](detectors/new_value.md): Detect new values in the variables in the logs.
* [Combo Detector](detectors/combo.md): Detect new combination of variables in the logs.

## Configuration

## Auto-configuration (optional)

Detectors can optionally support **auto-configuration** — a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually.

### Enabling auto-configuration

Auto-configuration is controlled by the `auto_config` flag in the pipeline config (e.g. `config/pipeline_config_default.yaml`):
When `auto_config` is set to `False`, the detector expects an explicit `events` block that specifies exactly which variables to monitor:

```yaml
detectors:
NewValueDetector:
method_type: new_value_detector
auto_config: True # enable auto-configuration
params: {}
# no "events" block needed — it will be generated automatically
auto_config: False
params: {} # global parameters
events: # event-specific configuration
1: # event_id
instance1: # name of instance (arbitrary)
params: {} # additional params
variables:
- pos: 0 # location of an unnamed variable from the log message
name: var1 # name of variable (arbitrary)
header_variables:
- pos: level # location of a named variable (defined in log_format of parser)
global: # define global instance for new_value_detector similar to "events"
global_instance1: # define instance name
header_variables: # same logic as header_variables in "events"
- pos: Status
```

When `auto_config` is set to `False`, the detector expects an explicit `events` block that specifies exactly which variables to monitor:

### Configuration semantics (preliminary)

**`events` key** — The integer key is the `EventID` (or `event_id`) to monitor (see the MatcherParser docs for how EventID is assigned).

**`variables[].pos`** — The 0-indexed position of the `<*>` wildcard in the matched template, counting from left to right starting at 0. For example, given:

```text
pid=<*> uid=<*> auid=<*> ses=<*> msg='op=<*> acct=<*> exe=<*> hostname=<*> addr=<*> terminal=<*> res=<*>'
```

`pos: 0` captures `pid=`, `pos: 6` captures `exe=`, etc.

**`header_variables[].pos`** — A named field from the log format string (e.g., `Type`, `Time`, `Content`) rather than a wildcard position.


### Auto-configuration (optional)

Detectors can optionally support **auto-configuration** — a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually.

Auto-configuration is controlled by the `auto_config` flag in the pipeline config (e.g. `config/pipeline_config_default.yaml`):

```yaml
detectors:
NewValueDetector:
method_type: new_value_detector
auto_config: False
auto_config: True # enable auto-configuration
params: {}
events:
1:
instance1:
params: {}
variables:
- pos: 0
name: var1
header_variables:
- pos: level
# no "events" block needed — it will be generated automatically
```


### How it works

When auto-configuration is enabled, the detector goes through two extra phases before training:
Expand Down
14 changes: 14 additions & 0 deletions docs/parsers/template_matcher.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,23 @@ The template matcher is a lightweight, fast parser intended for logs that follow
- Preprocesses logs and templates (remove spaces, punctuation, lowercase) based on config.
- Finds the first template that matches and extracts all wildcard parameters in order.
- Populates ParserSchema fields: `EventID`, `template`, `variables`, `logID`, and related fields.
- **`EventID` is the 0-indexed line number of the matched template** in the template file (first line → `EventID: 0`, second line → `EventID: 1`, etc.).

This parser is deterministic and designed for high-throughput use when templates are known in advance.

## EventID assignment (preliminary)

The `EventID` (or `event_id`) field in the output `ParserSchema` identifies which template was matched. It equals the **0-indexed line number** of the matching template in the template file:

| Line in template file | EventID |
|-----------------------|---------|
| 1st line | 0 |
| 2nd line | 1 |
| 3rd line | 2 |
| ... | ... |

This `EventID` is the integer key used in detector configurations (e.g., `NewValueDetector`) to scope detection rules to logs of a particular template type.

## Template format

- Templates are plain text lines in a template file.
Expand Down
Loading