Skip to content
BranchesTags

Repository files navigation

LogOcean — agentless SIEM: ingest, parse, normalize, store, detect, alert

LogOcean

tests python postgres agentless siem

A self-hosted, agentless SIEM — parse, normalize, detect, and retain security logs from 29 vendors in PostgreSQL.

LogOcean ingests network, endpoint, cloud, identity, and OT/ICS logs from many vendors through three front doors — web upload, an HTTP ingest API, and a syslog receiver (UDP/TCP/TLS) — then auto-detects the format, parses and normalizes each record to one common schema, indexes it for full-text and structured search, matches it against a Sigma-based detection & correlation engine plus threat-intel feeds, and retains everything in PostgreSQL for ≥ 3 years. Alerts flow into triage, cases, notifications, and agentless response — with UEBA, kill-chain reconstruction, a detection-coverage scoreboard, an AI copilot, and passive OT/ICS monitoring on top.

Table of contents

Overview

LogOcean is an agentless SIEM. Ingested events are matched against a Sigma-based detection & correlation engine and threat-intel IOC feeds, raising alerts you work at /alerts: triage them (assign, note, suppress noise) and group related ones into cases (/cases). New alerts are pushed to your channels and can trigger agentless response playbooks (audited at /responses). Collectors pull vendor / cloud / identity logs (Okta, GitHub, GitLab, AWS CloudTrail, Entra ID, Microsoft 365, GCP Cloud Audit Logs) while other tools push findings to the API.

UEBA entity-risk analytics (/risk) baseline every user / host / IP and flag new-entity / new-association anomalies beyond the rules; kill-chain reconstruction (/killchain) stitches related alerts across ATT&CK tactics into attack stories; and a detection workbench (/workbench) maps coverage, flags noisy / dead rules, and tests Sigma rules live. A /coverage scoreboard measures MITRE ATT&CK + ATLAS coverage, and a SigmaHQ importer brings in the community rule corpus. An optional AI SOC copilot (Claude) explains alerts, summarizes cases, and drafts Sigma rules from plain English. OT / ICS monitoring (/ot) ingests passive Zeek + ICSNPP telemetry (Modbus / DNP3 / S7comm / CIP …) with an ATT&CK-for-ICS rule pack, asset inventory, and master→controller baselining. Dashboards + /reports visualize it all (charts, top-N, ATT&CK-Navigator / CSV export), and auth / RBAC, an audit log, and a /compliance view (MITRE → NIST / CIS / ISO 27001 / SOC 2 / PCI / HIPAA + IEC 62443 / NERC CIP) round it out — all tested unit + integration against real PostgreSQL in CI.

Key features

  • 29 parsers, auto-detected across network / firewall, endpoint / IDS, cloud / identity, private cloud (Nutanix), OT / ICS, and generic CEF / LEEF / syslog / JSON.
  • One normalized schema (time, vendor, type, src/dst IP+port, user, host, action, severity, rule, bytes, message) with the full original record kept in jsonb.
  • Three ingest paths — web upload, HTTP API (API-key auth, gzip-aware), and a syslog receiver (UDP/TCP/TLS) fronted by a bounded async queue.
  • Sigma-based detection & correlation (90 detection + 10 correlation rules shipped) with the common Sigma modifiers, plus a community SigmaHQ importer.
  • CIM data models — 11 versioned, vendor-agnostic schemas (/datamodels) so a rule or a search binds to datamodels: web, not to a vendor. Data-not-code YAML registry.
  • Detection-coverage scoreboard for MITRE ATT&CK (Enterprise + ICS) and MITRE ATLAS, with Navigator-layer export and a CI rule-linter.
  • Threat-intelligence IOC enrichment, notifications (webhook / email), and agentless response playbooks (with time-boxed auto-revert).
  • Triage workflow — assignment, notes, suppression/allowlists, case grouping, and per-user saved searches on the event + alert views.
  • Investigation & analytics — UEBA entity risk, kill-chain reconstruction, a detection-engineering workbench, and an optional AI SOC copilot (Claude).
  • Passive OT / ICS monitoring (Zeek + ICSNPP) with an ATT&CK-for-ICS rule pack, asset inventory, and IEC 62443 / NERC CIP compliance mapping.
  • Agentless collectors for Okta, GitHub, GitLab, AWS CloudTrail, Entra ID, Microsoft 365, and GCP Cloud Audit Logs.
  • PostgreSQL storage, RANGE-partitioned by month, with GIN full-text, a jsonb GIN index, and btree indexes on the common fields — plus ≥ 3-year retention (purge = instant partition drop).
  • Auth / RBAC, security headers + CSRF, an audit log, and compliance reporting.

Architecture

Three inputs share one core: a source-agnostic detect → parse → normalize → store pipeline. Live sources buffer in a bounded async queue drained by writer workers, so a burst never blocks the receiver.

 upload (web) ───────────────┐
 POST /api/v1/ingest (key) ──┤─► auto-detect ─► parse ─► normalize ─► store (Postgres)
 syslog UDP/TCP/TLS ─► queue ┘     format                common      month-partitioned
                                                          schema      events + FTS index
                                                                           │
                                                  search ◄── filters + full-text ◄──┘

Every event is evaluated inline against per-event detection rules as it is stored; a background scheduler runs correlation (threshold) rules over the event store. Both raise rows in alerts, which fan out to notifications and response. The full original record is always kept in events.raw (jsonb) so nothing is lost and any field stays searchable.

  • Stack: Python 3.11–3.13, FastAPI + Uvicorn, Jinja2 (server-rendered UI), PostgreSQL 16 via psycopg 3 (+ psycopg_pool), python-dateutil.
  • No JS build step, no chart library — charts are server-rendered CSS/SVG.

Quick start

Docker

cp .env.example .env          # optional: adjust retention / limits
docker compose up --build     # starts Postgres + the app
# open http://localhost:8000

Then Upload a file (try the ones in samples/), and Search.

Two consoles, on purpose

The classic server-rendered console owns every page and every write, works with JavaScript disabled, and is what http://localhost:8000 gives you.

Alongside it, http://localhost:8000/ui/alerts is a React alert queue — the first screen of a client-side console, reading the same data through the JSON API at /console/api. It adds what a server-rendered page structurally cannot: client-side sort and narrowing with no round trip, row expansion, an optional live refresh, and every filter held in the URL so a view can be pasted into a ticket. It is a preview of one screen, not a replacement — nothing is being removed.

No Node is needed to run any of this. The bundle is committed to the repository (frontend/dist) so an air-gapped docker compose up --build needs no npm registry; a CI job rebuilds it from source and fails if the two disagree. The reasoning, and the two options rejected, are in docs/ADR-001. Working on the frontend needs Node 24: cd frontend && npm ci && npm run build, then commit the rebuilt dist with your change.

Local (without Docker)

python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# point at a Postgres you control:
export DB_DSN="postgresql://logocean:logocean@localhost:5432/logocean"   # PowerShell: $env:DB_DSN=...
uvicorn app.main:app --reload

The schema (tables, partitions, indexes) is created automatically on startup.

Supported log sources

All 29 parsers are auto-detected on upload (or selectable explicitly). Grouped by domain:

Domain Sources
Network / firewall Palo Alto NGFW (CSV export & syslog), Fortinet FortiGate, Cisco ASA / Firepower, Cisco IOS / IOS-XE / NX-OS, Cisco Meraki, Zeek (TSV & JSON)
Endpoint / IDS / host CrowdStrike Falcon (CSV & JSON), Windows Security Event Log, Microsoft Sysmon, Linux auditd, Apache / Nginx access logs, Suricata EVE
Cloud / identity AWS CloudTrail, Google Cloud Audit Logs, Microsoft Azure Activity, Microsoft 365 Unified Audit, Microsoft Entra ID sign-ins, Okta System Log, GitHub audit, GitLab audit
Private cloud / virtualization Nutanix Prism Central (audit / API / Flow), Nutanix Files / Data Lens (file-access audit)
OT / ICS (Zeek + ICSNPP) Modbus, DNP3, S7comm, CIP / EtherNet-IP, BACnet … enriched into a control-plane action + ot.* fields (see OT / ICS monitoring)
Generic CEF (ArcSight & many WAFs / proxies / AV), LEEF 1.0 / 2.0 (Tripwire, QRadar, Juniper, Check Point …), generic syslog (RFC 3164 / 5424), generic JSON / NDJSON (flat or ECS)

Every parser normalizes to one schema and keeps the full original record in jsonb.

How to export logs from each source
Source How to export Upload as
Palo Alto NGFW Monitor ▸ Logs ▸ (Traffic/Threat/URL/System/Config) ▸ Export to CSV Palo Alto CSV (auto)
Palo Alto NGFW Syslog file from your collector / forwarder Palo Alto syslog (auto)
Fortinet FortiGate Syslog from your collector, or FortiAnalyzer ▸ Log download Fortinet FortiGate (auto)
CrowdStrike Falcon Endpoint security ▸ Detections / Incidents ▸ Export (CSV) CrowdStrike CSV (auto)
CrowdStrike Falcon Event Search / API / FDR export (JSON or NDJSON) CrowdStrike JSON (auto)
Windows hosts Get-WinEvent -LogName Security ▸ Export-Csv (or ConvertTo-Json); or Event Viewer ▸ Save All Events As CSV Windows Security (auto)
Windows hosts (Sysmon) Get-WinEvent -LogName 'Microsoft-Windows-Sysmon/Operational' ▸ ConvertTo-Json / Export-Csv Sysmon (auto)
Linux hosts /var/log/audit/audit.log (auditd) Linux auditd (auto)
Apache / Nginx access.log (Common / Combined Log Format) Apache / Nginx access (auto)
Suricata IDS/IPS eve.json (NDJSON) or an exported JSON array Suricata EVE (auto)
Cisco ASA / Firepower Syslog from your collector (lines with %ASA-…/%FTD-…) Cisco ASA / Firepower (auto)
Cisco IOS / IOS-XE / NX-OS Device syslog (lines with %FACILITY-SEV-MNEMONIC) Cisco IOS (auto)
Cisco Meraki Dashboard ▸ syslog server output (flows / urls / ids-alerts …) Cisco Meraki (auto)
Zeek (Bro) conn.log / dns.log / http.log — classic TSV (#fields) or JSON Zeek TSV / JSON (auto)
AWS CloudTrail S3/CloudWatch export or aws cloudtrail lookup-events (JSON) AWS CloudTrail (auto)
Google Cloud Cloud Logging export or gcloud logging read --format json Google Cloud Audit (auto)
Microsoft Azure Activity Log export ({"records":…}) or az monitor activity-log list Microsoft Azure Activity (auto)
Microsoft 365 Search-UnifiedAuditLog ▸ AuditData, or Management Activity API (JSON) Microsoft 365 (auto)
Microsoft Entra ID Sign-in logs via Graph auditLogs/signIns or Azure Monitor export (JSON) Microsoft Entra ID (auto)
Okta System Log API export (JSON array / NDJSON) Okta System Log (auto)
GitHub Org/Enterprise ▸ audit log ▸ Export (JSON / NDJSON) GitHub audit (auto)
GitLab Admin ▸ /audit_events API (JSON) GitLab audit (auto)
Nutanix Prism Central Settings ▸ Syslog server (modules: Audit / API Audit / Flow) → your collector Nutanix Prism Central (auto)
Nutanix Files / Data Lens Partner server (vendor_name: syslog, :1468) file-audit notifications, or a File-Analytics / Data-Lens JSON export Nutanix Files (auto)
Any CEF source Syslog / file in Common Event Format (CEF:0|…) CEF (auto)
Tripwire Log Center / Enterprise Forwarder ▸ send events as LEEF or CEF (or the exported log file) LEEF / CEF (auto)
IBM QRadar Routing/forwarding rule ▸ export events as LEEF (its native format), or CEF / syslog. (A QRadar system-backup archive is proprietary — export events first.) LEEF (auto)
Any LEEF source (QRadar, Juniper, Check Point …) Syslog / file in Log Event Extended Format (LEEF:1.0|… / LEEF:2.0|…) LEEF (auto)
Any syslog source Plain RFC 3164 / 5424 syslog not matched above Generic syslog (auto)
Any JSON source Flat or ECS-style JSON / NDJSON not matched above Generic JSON (auto)

Auto-detect inspects the header/content; if a file is ambiguous, pick the format explicitly in the upload form.

Live ingestion

Besides manual upload, LogOcean accepts logs in near-real-time through two front doors that share the same detect → parse → normalize → store pipeline.

HTTP ingest API

Create a key on the Admin page, then POST raw log content:

curl -X POST "http://localhost:8000/api/v1/ingest?format=auto&filename=fw.log" \
     -H "X-API-Key: lo_..." --data-binary @fw.log
# -> {"batch_id": 42, "format": "...", "total": N, "inserted": N, ...}

format may be auto or any format key; auth is X-API-Key or Authorization: Bearer. Only the sha256 of each key is stored (the plaintext is shown once). A gzip body is decompressed transparently (send .gz bytes directly, or --data-binary @fw.log.gz) — the MAX_UPLOAD_MB limit then applies to the decompressed size, with a decompression-bomb guard. Drag-dropping a .gz file into the web Upload page works the same way.

Syslog receiver

Set SYSLOG_ENABLED=true to listen on UDP+TCP (default port 5514; TLS optional). Point a collector or device at it:

logger -n localhost -P 5514 -d "<134>1 2026-06-24T10:00:00Z fw test message"

Messages flow through a bounded async queue with writer workers that batch-insert, so a burst never blocks the receiver; queue counters are on GET /health. TCP framing supports both octet-counting (RFC 6587) and newline-delimited streams.

Live ingest can lose events, and /health says so. If a flush fails (typically the database is unreachable) the buffered group is discarded — it exists only in memory and syslog has no upstream to replay from. /health reports events_lost alongside flush_errors (the latter counts incidents, so one failed flush of a full buffer read as 1) and returns status: degraded while either is non-zero. dropped is the different failure of arriving faster than the writer drains. A durable dead-letter spool is not implemented.

Bulk import (large / historical backups)

To load a big multi-GB export — e.g. a 3-year IBM QRadar LEEF backup — use clients/logocean_import.py. It streams the file in size-bounded, line-aligned chunks (so no single request exceeds MAX_UPLOAD_MB), decompresses .gz locally, retries transient failures, and prints progress. Ingest is idempotent (per-event dedup hash), so re-running after an interruption is safe.

python clients/logocean_import.py --url http://localhost:8000 --key lo_... \
    --format leef --max-mb 400 qradar_3yr.leef.gz
# [chunk 1] 400.0 MB -> inserted=812345 duplicates=0 (batch 5)  ...

Events keep their original timestamps, so 3 years of history lands in the correct monthly partitions (not the upload date). For a historical backfill, consider DETECTION_ENABLED=false and UEBA_ENABLED=false during the load to avoid a flood of stale alerts and to speed ingest. --gzip compresses each POST body (the server gunzips it) to save bandwidth; --max-mb must be ≤ the server's MAX_UPLOAD_MB. Note: a proprietary QRadar system backup archive is not readable — export the events as LEEF/CEF/syslog first.

Search & analytics (LOQL)

LOQL is LogOcean's piped search/transform language (Splunk-SPL-shaped) — the first backbone of the Splunk-class transformation roadmap. A query is a pipeline of stages separated by |, and it compiles to parameterized SQL over the partitioned event store:

vendor=paloalto action=deny _time > "-24h" | stats count as n by src_ip | sort -n | head 10

Batch 1 ships search · where · eval · fields · rename · sort · head · dedup · stats · top · rare · bin · timechart, with stats/timechart functions (count, sum, avg, min, max, dc, values, list), schema-on-read on the raw JSON (errorCode=0 reads a field no column exists for), relative time (_time > "-24h"), and wildcard (host="web*") matching. Run it via POST /api/v1/query (API-key auth):

curl -X POST http://host:8000/api/v1/query -H "X-API-Key: lo_..." \
     -H "Content-Type: application/json" \
     -d '{"query": "vendor=fortinet action=deny | top 10 src_ip", "limit": 500}'

Safe by construction: every value, jsonb key, and glob you type is a bound parameter — the compiler (app/loql/) emits only fixed SQL skeletons, so a malformed or hostile query is a clean 400, never SQL injection. Guardrails (LOQL_TIMEOUT_MS, LOQL_MAX_ROWS, LOQL_DEFAULT_LIMIT) stop one heavy query from starving ingest on the shared Postgres. Full reference: docs/LOQL.md.

CIM data models

CIM is LogOcean's versioned, vendor-agnostic data-model layer — the second backbone of the transformation roadmap. A data model is a named schema plus a membership rule that decides which events belong to it, so a search or a detection binds to what an event is, not who made the box:

| datamodel Authentication | stats count by user, action | sort -count   # LOQL
datamodels: web                                                          # a detection rule
SELECT * FROM cim_ics WHERE is_write = 'true'                            # SQL view

Eleven models ship — Authentication · Network · Web · DNS · Endpoint · Change · Malware · IDS · Industrial (OT/ICS) · Email · Vulnerability — nine of them populated by the parsers you already have (Email and Vulnerability are defined and waiting for a mail gateway / scanner source, and say so). Membership is many-to-many: an OT session is ['ics', 'network'], a Falcon detection is ['endpoint', 'malware']. Browse them all at /datamodels, with an opt-in member count per model.

Membership is materialized on the row: events.cim_models is a plain text[], GIN-indexed, filled per event in Python at ingest — exactly the way search_tsv is filled — so cim_models @> ARRAY['web'] is an index lookup, and detection can see model membership before the row is inserted. Deliberately not a generated column: PostgreSQL 16 freezes a generation expression at ADD COLUMN (rewriting one needs ALTER COLUMN … SET EXPRESSION, PG17+, and we pin postgres:16), which would have made every future registry edit unreachable on an existing database.

The whole layer is data, not code — app/cim/models.yaml. Add a source to a model, or add a model, by editing YAML: no Python, no schema migration. A field mapping may read a normalized column, a constant, or one or several jsonb keys (vendors spell the same concept differently), including nested paths:

- name: Authentication
  tag: authentication
  membership:
    - {log_type: [signin, authentication, auth, authpriv]}
    - vendor: [microsoft]
      product: [windows]
      log_type: [security]
      event_id:                                  # ordered alternatives -> COALESCE
        raw: [event_id, EventID, "Event ID", Id, [Event, System, EventID]]
        values: [4624, 4625, 4768, 4769, 4776]
  fields:
    - {name: user, column: user_name}
    - {name: src,  column: src_ip}
    - {name: query, raw: [query, QueryName, [dns, rrname]]}

Operator note — every edit needs a restart. models.yaml is data, but nothing in it is picked up by a running process: the registry is parsed and validated once and cached for the process lifetime. A fields: edit takes effect on the next restart (the cim_<tag> views are rebuilt, and nothing stored needs correcting — a field is computed on read). A membership: edit also takes effect on the next restart, and then only for new events; history is corrected by running Admin → Backfill membership.

Restart first, then backfill — in that order. A backfill run before the restart re-derives every row under the old cached rule, changing nothing while stamping the progress marker, so /admin would report history as current under a rule that had never been applied to a single row. /admin shows a restart required banner when the file on disk has drifted from the loaded registry, and a backfill due banner when stored history predates the current rule. Full reference: docs/CIM.md.

Detection & alerting

Every ingested event is evaluated against detection rules, and a background scheduler runs correlation rules over the event store. Matches raise alerts you can filter, drill into (down to the originating event), and triage at /alerts; open-alert counts show on the dashboard.

Rules

Rules are YAML files in rules/ (a Sigma-compatible subset), tagged with MITRE ATT&CK, and enable/disable from the Admin page (applies immediately). The engine supports the common Sigma field modifiers so many community rules load as-is: contains/startswith/endswith/re (with i/m/s flags)/cased, value lists with |all, cidr (IP-in-network), numeric lt/lte/gt/gte, exists, fieldref, and the base64/base64offset/windash encoding modifiers for command-line obfuscation — plus the and/or/not + N of … condition grammar.

# per-event rule
title: RDP Connection Allowed
id: lo-rdp-allowed
level: medium
fidelity: medium            # high | medium | hunt — drives the coverage scoreboard
data_source: [firewall]     # ATT&CK-style data-source key(s)
detection:
  selection: { dst_port: 3389 }
  permitted: { action: [allow, accept] }
  condition: selection and permitted
tags: [attack.t1021.001, attack.lateral_movement]

A per-event rule can also bind to a CIM data model instead of (or as well as) a vendor — datamodels: web fires on Apache, Nginx, Zeek, Suricata, PAN URL-filtering, Meraki, FortiGate webfilter and CEF proxy alike, because the model owns that list:

title: SQL Injection Attempt
id: lo-web-sqli
datamodels: web             # a tag, a display name, or a list (ANY of them)
detection:
  sel: { message|contains: ["union select", "' or 1=1"] }
  condition: sel

datamodels: and logsource: are ANDed; an omitted binding is match-all (so every pre-existing rule is unchanged and costs nothing); and a name that resolves to no model disables that rule rather than widening it — the CI rule-linter fails on it. See docs/CIM.md.

# correlation (threshold) rule — e.g. brute force
title: Brute Force - Failed Logon Burst
id: lo-corr-bruteforce-logon
level: high
correlation:
  match: { action: failed-logon }
  group_by: [src_ip]
  window: 5m
  threshold: 5
tags: [attack.t1110, attack.credential_access]

Correlation rules can also count distinct values of a second column instead of raw events (distinct_count:), which expresses behaviours defined by breadth — e.g. password spraying (one source failing against many distinct accounts) rather than a simple failed-logon burst:

# cardinality rule — one src_ip → 10+ distinct accounts = password spray
title: Password Spray - One Source, Many Accounts
id: lo-corr-password-spray
level: high
correlation:
  match: { action: failed-logon }
  group_by: [src_ip]
  distinct_count: user_name      # count DISTINCT users, not raw failures
  window: 10m
  threshold: 10
tags: [attack.t1110.003, attack.credential_access]

LogOcean ships a starter pack of 90 detection + 10 correlation rules covering failed-logon brute force, denied-connection floods, RDP exposure (incl. external RDP via cidr), ingress-tool transfer, event-log clearing, security-tool tampering, encoded/download PowerShell (base64offset/windash), AWS CloudTrail (logging disabled, root console login, world-open security groups, access-key creation), Entra ID (risky sign-in succeeded, legacy auth), Okta (admin grant, MFA deactivation), Microsoft 365 (mailbox forwarding rules), GitHub (repo made public), and these purpose-built packs:

  • Tripwire file-integrity monitoring — critical system / credential file change, web-shell drop, persistence-mechanism change, integrity monitoring disabled, monitored-object deletion, plus a mass-change-burst correlation for ransomware / bulk tampering.
  • Sysmon / endpoint — Office spawning a shell, LOLBin proxy execution, registry Run-key / WMI persistence, LSASS credential dumping, shadow-copy deletion, scheduled-task creation, command-line log clearing, plus a curated high-fidelity endpoint pack (Phase 2): LSASS memory access by mask, known credential-dumper tools, NTDS/SAM extraction, WDigest plaintext-caching, BYOVD vulnerable-driver load, Defender-disable & AMSI/ETW tampering, UAC registry hijack, LSA/AppInit persistence, remote-thread injection, PsExec / msiexec / BITS execution, Cobalt Strike named pipes, and local-account creation — each fidelity-tagged and IOC-verified.
  • Cloud & identity (Phase 3) — a high-value pack across every cloud/identity source, adding first-time coverage for GCP, Azure and GitLab: AWS (threat-detection / Config disabled, admin-policy attach, public-S3 exposure), GCP (privileged IAM grant, service-account key created, logging-sink deleted), Azure (RBAC role assignment, diagnostic-settings deleted, Key Vault tamper), GitLab (2FA disabled, project made public), Okta (admin impersonation), Microsoft 365 (org transport rule, mailbox delegate), GitHub (org 2FA disabled, branch-protection removed) — every API event/operation name adversarially verified against provider docs.
  • Linux & network / web-exploitation (Phase 4) — request-line detections for web attacks (T1190): SQL injection, path traversal / LFI, OS command injection, XSS, web-shell access, and scanner user-agents (each fires across Apache/Nginx, Zeek HTTP and Suricata HTTP); Suricata IDS passthrough (web-app-attack, trojan/C2, crypto-mining categories); and Linux auditd TTPs — reverse shells, setuid backdoors, sudoers / SSH-authorized_keys / cron persistence, /etc/shadow access, and security-control disabling — with every payload pattern and IDS category string adversarially verified.
  • Behavioural correlation (Phase 5) — beyond simple event-count bursts, cardinality (distinct-count) rules catch behaviours defined by breadth: password spray (one source → many distinct accounts), distributed brute force (one account ← many distinct source IPs), port scan (one host → many distinct ports), and network sweep (one source denied to many distinct hosts).
  • OT / ICS — Modbus write / diagnostic, S7comm program download / PLC stop, DNP3 device restart / disable-unsolicited, CIP set-attribute write, an IT→OT write conduit-violation (Purdue / IEC 62443 zone) rule, and an OT-protocol enumeration correlation — tagged with ATT&CK for ICS (see OT / ICS monitoring).
  • Nutanix Prism Central — VM / cloud-instance deletion via the REST API, cluster unregister / detach, user / role / authentication change, plus a Flow microsegmentation drop-burst correlation for internal scanning / lateral movement.
  • Nutanix Files / Data Lens — ransomware-encrypted-extension / ransom-note write, share ACL / permission change, plus a mass-file-deletion-per-user correlation for ransomware / wiper / insider destruction.

Detection can be turned off with DETECTION_ENABLED=false.

Triage & tuning

Each alert can be acknowledged / closed / reopened, assigned to an analyst, and annotated with threaded notes. To cut noise, build suppression / allowlist rules — by rule id, source IP/CIDR, user, host and/or vendor (each set field is an AND condition). A matching alert is stored as suppressed (kept for audit, hidden from the default queue, and never notified or actioned), and the suppression's hit count is tracked. The fastest way to make one is the Suppress similar form on any alert (pre-filled from its attributes); manage them all under Admin ▸ Suppressions.

Cases / incidents

Group related alerts into one investigation at /cases. Create a case from any alert (or add it to an open one), give it a status (open / investigating / closed), an assignee and threaded notes; its severity rolls up to the highest of its member alerts. The case page suggests related alerts — open, un-cased alerts sharing a source IP, user or host with the case — so a burst of activity folds into a single timeline with one click.

Saved searches

Name and re-run your common queries. On both /search and /alerts, a Save current box stores the active filter set; saved searches appear as one-click chips above the results (delete with the ×). They are per-user (when auth is on) and scoped to their page, so "my open high alerts" or "Fortinet denies this week" is always one click away.

Detection coverage

Coverage scoreboard

The /coverage page grows the detection pack against a measured MITRE ATT&CK (Enterprise + ICS) and MITRE ATLAS (adversarial-AI) coverage map, computed from the rules themselves — what LogOcean can detect, not what has fired. It rolls coverage up by tactic, fidelity (high / medium / hunt) and data source, renders the ATLAS matrix (a scaffold until AI/LLM telemetry lands), and exports MITRE ATT&CK Navigator layers scored by rule coverage (/coverage/attack-navigator.json, ?domain=ics for the ICS matrix). A CI rule-linter enforces per-rule quality (unique ids, valid level/fidelity, an attack.*/atlas.* tag). The programme is tracked in docs/DETECTION_COVERAGE_ROADMAP.md.

Import community SigmaHQ rules

scripts/import_sigma.py translates the thousands of open, ATT&CK-tagged SigmaHQ rules into LogOcean's engine — Sigma logsource becomes a gate over our normalized fields (a process_creation rule only fires on process-create events), Sigma field names pass through (our Sysmon/endpoint parsers lift them onto raw), and anything we can't faithfully run is skipped with a reason (unmapped logsource, unsupported modifier, Sigma aggregation/correlation, deprecated). Point it at a clone and it emits rules/imported/*.yml (loaded alongside rules/, generated per-deployment):

git clone https://github.com/SigmaHQ/sigma
python scripts/import_sigma.py --src sigma/rules --write   # then restart to load

Notifications & response

Newly-raised alerts at or above NOTIFY_MIN_LEVEL are delivered to notification channels — a webhook (Slack/Teams/Discord/generic, WEBHOOK_URL) and/or email (SMTP_*) — by a background worker, so slow delivery never blocks ingest. Set NOTIFY_ENABLED=true and configure a channel.

Alerts can also trigger response playbooks (playbooks/*.yml). A playbook matches alerts (by rule id / severity / technique) and runs an agentless action: a structured webhook POST to your automation/SOAR/firewall/IAM endpoint (RESPONSE_WEBHOOK_URL) carrying the intent, or a log action that only records. LogOcean stays agentless and lets your platform enforce. Every action is audited at /responses and on the alert's page.

A revert_after makes the action time-boxed: a background scheduler (RESPONSE_AUTO_REVERT, on by default) fires the inverse intent (block_ip→unblock_ip, disable_user→enable_user, …) once it expires, so temporary containment lifts itself — the revert appears as its own /responses row.

# playbooks/block_bruteforce_source.yml
title: Block brute-force source IP
id: pb-block-bruteforce
match: { rule_id: [lo-corr-bruteforce-logon], min_level: high }
action: { type: block_ip, target: src_ip }   # POSTs {action, target, alert} to your SOAR
revert_after: 600                             # auto-POST unblock_ip after 10 min

UEBA & entity risk

Beyond signature rules, LogOcean tracks behaviour. With UEBA_ENABLED (on by default), every ingested event updates lightweight entity baselines — one row per user / host / source IP with first-seen / last-seen, plus the associations between them (user↔IP, user↔host, host↔IP). The /risk page then shows:

  • Riskiest entities (users, hosts, IPs), scored from their attributed alerts — severity-weighted and recency-decayed (RISK_HALF_LIFE_DAYS), so a recent critical outweighs an old low. Click through to a per-entity timeline, associations, and alerts (/entity?etype=user&value=…).
  • Anomalies (24h) — new entities (first time we've ever seen this user/host/IP) and new associations (an established actor suddenly using a new IP or host — a classic account-takeover / lateral-movement signal), surfaced without any rule having to fire.

It's pure PostgreSQL (no ML dependency); the baselines are maintained incrementally in the ingest path.

Log-source health

A SIEM is blind to what it stops receiving. SOURCE_HEALTH_ENABLED (on by default) watches every log source — identified by vendor/log_type — and raises an alert when one that was sending regularly goes silent. That covers a broken collector or forwarder, a device that fell over or was reconfigured, and the case that matters most for security: an attacker disabling logging to hide their tracks (MITRE ATT&CK T1562 — Impair Defenses).

The /sources page lists each source with its last-seen age and a healthy / silent status. A source becomes "expected" once it has sent at least SOURCE_HEALTH_MIN_EVENTS in the last SOURCE_HEALTH_LEARN_DAYS; it is flagged silent when its newest event is older than SOURCE_HEALTH_SILENCE_MINUTES. A background check runs every SOURCE_HEALTH_INTERVAL seconds, and an ongoing outage re-alerts at most once per SOURCE_HEALTH_REPEAT_HOURS — the alert flows to notifications and response playbooks like any other. It's stateless (derived from the event store each run, no extra table) and the assessment logic is pure.

Kill-chain reconstruction

A single alert is one step; an intrusion is a sequence of steps by the same actor across ATT&CK tactics over time. The /killchain page stitches related alerts into attack stories:

  • alerts are linked when they share an entity (user / host / IP) and fall within KILLCHAIN_MAX_GAP_MINUTES of each other (single-linkage, so a long campaign chained through intermediate alerts stays one story);
  • a linked group only qualifies when it spans ≥ KILLCHAIN_MIN_TACTICS distinct ATT&CK tactics — showing progression along the kill chain, not just a burst of one behaviour;
  • each story is presented as kill-chain-ordered stages (Initial Access → Execution → Credential Access → …), the pivot entities that tie it together, a rolled-up severity, and a plain narrative.

Promote a story to an investigation case with one click (severity + alert linkage roll up exactly like a manually built case). With KILLCHAIN_AUTOCREATE=true, a background scheduler auto-promotes stories at or above KILLCHAIN_MIN_SEVERITY, de-duplicated by the story's alert-set signature. The reconstruction runs over recent un-cased alerts, so once a story is folded into a case it won't be re-surfaced. Like UEBA, it's pure PostgreSQL — the reconstructor (app/killchain.py) is dependency-free and fully unit-tested.

Detection-engineering workbench

The /workbench page helps you tune the detection pack itself:

  • ATT&CK coverage map — per-tactic (kill-chain-ordered) view of which techniques the enabled rules cover, and the gaps (techniques only a disabled rule would catch, or none at all), plus an overall coverage %.
  • Rule health — every rule with its firing counts over the last WORKBENCH_WINDOW_DAYS, bucketed into noisy (≥ WORKBENCH_NOISY_THRESHOLD alerts in the window), never-fired (enabled but no alert on record — untested or dead), and stale (fired historically, silent now).
  • Rule tester — paste a Sigma-subset rule and a sample event and evaluate it with the same engine the pipeline uses; the result shows the final verdict, the logsource match, and each named selection's boolean so you can see exactly why it did or didn't fire. Event fields may be normalized names (user_name, src_ip…) or raw vendor fields.

The analytics (app/workbench.py) are pure functions over the rule registry, so they're fully unit-tested without a database.

AI SOC copilot

With COPILOT_ENABLED=true and an API key, LogOcean adds Claude-powered assistance at three points (off by default; the app runs fine without the anthropic package):

  • Alert explainer — on any alert, Explain this alert sends the alert plus related activity (same entity) to Claude and returns a plain-language "what happened / why it fired / severity & likelihood / triage steps" briefing.
  • Case summarizer — on a case, Summarize this case drafts an incident summary (attack narrative in ATT&CK order, impacted entities, recommended actions) from the member alerts and notes; one click saves it as a case note.
  • Sigma-from-natural-language — on the workbench, describe a detection in English and Claude drafts a Sigma-subset rule, loaded straight into the rule tester so you can validate it before adding it to rules/.

The model is configurable (COPILOT_MODEL, default claude-opus-4-8) so operators can trade cost for capability (e.g. claude-sonnet-4-6). Prompt construction and the Sigma-extraction logic (app/copilot/prompts.py) are pure and fully unit-tested; the SDK call is a thin wrapper that degrades gracefully when unconfigured. Every AI action is RBAC-gated (analyst) and written to the audit log.

OT / ICS monitoring

LogOcean monitors operational-technology networks — PLCs, RTUs, and other controllers — the safe way: 100% passive and agentless. It never talks to a device or issues a control command; it ingests copies of telemetry produced by a network sensor, exactly the posture OT requires.

The telemetry source is Zeek running the ICSNPP (Industrial Control Systems Network Protocol Parsers, from CISA / INL) analyzers, which deep-packet-inspect the OT protocols and write modbus.log / dnp3.log / s7comm.log / cip.log … in the usual Zeek #fields shape. LogOcean's Zeek parsers enrich those records:

  • the control-plane operation is lifted onto the normalized action (write-registers, write-coils, diagnostic, program-download, plc-stop, cold-restart, disable-unsolicited, write-attribute, …), and log_type is set to the canonical protocol (modbus / dnp3 / s7comm / cip / enip / …);
  • an ot.* field set (ot.protocol, ot.operation = read/write/control, ot.is_write, ot.function_code, ot.unit_id, ot.address, …) is added to the event's raw, so every OT attribute is searchable and rule-matchable.

On top of that, the shipped OT rule pack flags the dangerous operations, tagged with ATT&CK for ICS techniques:

Rule Technique
Modbus write command to a controller T0855 / T0836
Modbus diagnostic / restart function T0814
S7comm program / block download to a PLC T0843 / T0889
S7comm PLC stop / operating-mode change T0858 / T0813
DNP3 device restart (cold / warm) T0816
DNP3 disable unsolicited reporting T0878
CIP set-attribute (write) to a device T0855 / T0836
IT→OT write/control across a zone boundary (Purdue / IEC 62443) T0855 / T0886
OT-protocol enumeration from one source (correlation) T0846

ATT&CK for ICS is wired through the rest of the platform too: kill-chain reconstruction understands the ICS tactics (inhibit-response-function, impair-process-control, …) in a merged IT→OT kill chain, so an intrusion that starts on IT and ends in process impact stitches into one attack story; and the Navigator export serves an ICS layer at GET /reports/attack-navigator.json?domain=ics-attack.

OT analysis (/ot). A dedicated page turns the OT telemetry into operator analytics: an asset inventory of the controllers (PLCs / RTUs) seen on the wire — the protocols each speaks, how many masters talk to it, and its control-op volume; a master → controller conversation map that flags a new-writer (a source first seen in the last 24h already issuing write / control commands to a controller — the top OT signal, an unexpected engineering client); and read / write / control activity per protocol. It's pure PostgreSQL aggregation over the events, no schema change. The IT→OT conduit-violation rule (rules/ot_it_to_ot_write.yml) enforces the Purdue / IEC 62443 zone model with cidr selections — a write/control command from outside the OT zone alerts (tune the CIDRs to your control network).

Compliance. OT rule coverage maps to IEC 62443-3-3 System Requirements and NERC CIP (plus NIST 800-53) on the /compliance page, alongside the existing IT frameworks.

Scope & limits. LogOcean is a log platform, not a sensor — it needs Zeek+ICSNPP (or a commercial DPI sensor) on the wire; it does no DPI itself. Serial Modbus and L2 GOOSE / Sampled-Values can't arrive over IP logs and need a sensor that sees them. OT response stays passive: alert / ticket / enforce at the IT boundary — never a device command.

Agentless collectors & feeds

Two agentless ways to get logs in without manual upload:

  • Pull collectors — set COLLECTORS_ENABLED=true and a collector's credentials. A scheduler fetches new records every COLLECTOR_INTERVAL seconds, checkpointing a per-source cursor so each run only pulls what's new, and feeds them through the same parse→detect→alert pipeline. Status + enable/disable are on the Admin page. Built-in collectors (each activates only when its credentials are set):

    • Okta System Log (OKTA_*), GitHub audit log (GITHUB_*), GitLab audit events (GITLAB_*) — token-based REST.
    • AWS CloudTrail (AWS_REGION + AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN) — LookupEvents signed with AWS SigV4 (stdlib).
    • Microsoft Entra ID sign-in logs and Microsoft 365 unified audit log — one Azure app registration (AZURE_TENANT_ID/AZURE_CLIENT_ID/AZURE_CLIENT_SECRET), OAuth2 client-credentials. Entra uses Microsoft Graph; M365 uses the Office 365 Management Activity API and additionally needs M365_ENABLED=true.
    • GCP Cloud Audit Logs (GCP_PROJECT_ID + GCP_CLIENT_EMAIL + GCP_PRIVATE_KEY from a service-account key) — Cloud Logging entries:list, authenticated with a service-account signed-JWT OAuth2 grant (RS256 signed via stdlib — no SDK). Grant the service account roles/logging.viewer.

  • Push feeds — point your own tools (RHEL/Windows/SBOM/AWS audit scanners) at the ingest API. Copy clients/logocean_push.py:

    from logocean_push import push
push("http://logocean:8000", "lo_...", findings)   # list of dicts -> generic_json

Threat-intelligence enrichment

Set THREATINTEL_ENABLED=true to match every ingested event against indicators of compromise — IPs, CIDRs, domains, file hashes, and URLs. A hit raises a Threat Intelligence Match alert that flows through the normal alert → notify → response path (its severity is the highest of the matched indicators).

  • Feeds — THREATINTEL_FEEDS is a comma/space-separated list of local file paths or http(s) URLs, refreshed every THREATINTEL_REFRESH_MINUTES. Each feed may be a plain one-indicator-per-line list (# comments allowed), a CSV (indicator[,type[,severity[,description]]]), or JSON (an array of strings or of objects). Indicator type is inferred when not given.
  • Manual indicators — add or remove individual IOCs from the Admin page; a Reload button re-fetches the feeds.
  • Matching — indicators are held in an in-memory index for fast per-event lookup; the matcher checks the event's src/dst IPs (exact + CIDR) and pulls IPs/domains/URLs/hashes out of the normalized fields and raw (and from free text in message), so an indicator buried in a log line is still caught.
# a feed can be as simple as a blocklist file:
echo -e "# bad infra\n203.0.113.5\nevil.example\nhttp://drop.example/x" > feeds/iocs.txt
THREATINTEL_ENABLED=true THREATINTEL_FEEDS=feeds/iocs.txt  # ...then run as usual

Dashboards & reporting

The dashboard (/) shows headline counters (events, open alerts, open cases, on-disk size), an alert-volume time series, and top-N breakdowns — top firing rules, top MITRE ATT&CK techniques, and top alert/event source IPs — plus the existing per-vendor / per-log-type / partition tables. All charts are server-rendered CSS/SVG (no JS chart library, no extra dependencies).

The /reports page is a print-friendly summary over a selectable period (7–90 days) — headline metrics, alert status, the same time series and top-N charts, and the coverage span. Use Print / Save as PDF for a shareable PDF, and the toolbar buttons to export:

  • ATT&CK Navigator layer — GET /reports/attack-navigator.json?days=N returns a Navigator (layer 4.5) JSON scoring each technique by alert volume; load it at the ATT&CK Navigator to visualize coverage. Add &domain=ics-attack for an ATT&CK for ICS layer (OT T0NNN techniques); the default is the Enterprise matrix.
  • Alerts CSV — GET /alerts.csv streams the alert list (honours the /alerts filters), with spreadsheet-formula injection neutralized.

Configuration

All settings are environment variables (see .env.example).

Variable Default Meaning
DB_DSN postgresql://logocean:logocean@localhost:5432/logocean PostgreSQL connection
RETENTION_YEARS 3 Retention floor; purge cannot go below this
PAGE_SIZE 100 Search results per page
MAX_UPLOAD_MB 512 Reject larger uploads / API payloads
AUTO_PURGE false If true, drop partitions older than RETENTION_YEARS on startup
DETECTION_ENABLED true Evaluate detection + correlation rules and raise alerts
CORRELATION_INTERVAL 60 Seconds between correlation-rule evaluations
NOTIFY_ENABLED / NOTIFY_MIN_LEVEL false / high Send new alerts (>= level) to channels
WEBHOOK_URL / WEBHOOK_STYLE — / slack Notification webhook (Slack-text or full JSON)
SMTP_HOST … SMTP_TO — Email notification channel (host + from + to to activate)
RESPONSE_ENABLED / RESPONSE_WEBHOOK_URL false / — Run response playbooks; automation endpoint
RESPONSE_AUTO_REVERT / RESPONSE_REVERT_INTERVAL true / 60 Auto-undo time-boxed actions; sweep period (s)
COLLECTORS_ENABLED / COLLECTOR_INTERVAL false / 300 Scheduled pull collectors; poll period (s)
OKTA_* / GITHUB_* / GITLAB_* / AWS_* / AZURE_* / GCP_* — Per-collector credentials (a collector activates when set)
THREATINTEL_ENABLED / THREATINTEL_FEEDS false / — Match events against IOC feeds (paths or URLs)
THREATINTEL_REFRESH_MINUTES / THREATINTEL_DEFAULT_SEVERITY 60 / high Feed refresh period; severity when a feed omits one
UEBA_ENABLED true Maintain entity baselines + the /risk page (behavioural analytics)
RISK_HALF_LIFE_DAYS / RISK_WINDOW_DAYS 7 / 30 Risk-score decay half-life; scoring window
KILLCHAIN_ENABLED true Reconstruct multi-tactic attack stories on the /killchain page
KILLCHAIN_WINDOW_HOURS / KILLCHAIN_MAX_GAP_MINUTES / KILLCHAIN_MIN_TACTICS 24 / 60 / 2 Look-back window; max gap to link alerts; min distinct tactics for a story
KILLCHAIN_AUTOCREATE / KILLCHAIN_INTERVAL / KILLCHAIN_MIN_SEVERITY false / 300 / high Auto-promote high-severity stories to cases; poll period (s); severity floor
WORKBENCH_WINDOW_DAYS / WORKBENCH_NOISY_THRESHOLD 30 / 50 Rule firing-stats window; alerts/window above which a rule is flagged noisy
LOQL_DEFAULT_LIMIT / LOQL_MAX_ROWS / LOQL_TIMEOUT_MS 1000 / 100000 / 30000 LOQL guardrails: default result LIMIT, hard row cap, per-query statement_timeout
LOQL_MAX_AGG_ELEMS 10000 Cap on elements one values()/list() aggregate may collect (the row cap counts rows, not cells)
CIM_ENABLED true Rebuild the per-model cim_<tag> views on startup. Gates the views only — membership tagging at ingest is not optional
CIM_COUNT_DAYS / CIM_COUNT_TIMEOUT_MS 30 / 5000 /datamodels member counts: window (0 disables) and the budget for the whole sweep
CIM_BACKFILL_CHUNK 2000 Rows per committed chunk of the Admin → membership backfill
COPILOT_ENABLED false Enable the Claude-powered alert/case explainers + Sigma-from-NL (needs anthropic + a key)
COPILOT_API_KEY / COPILOT_MODEL / COPILOT_MAX_TOKENS — / claude-opus-4-8 / 1024 API key (else ANTHROPIC_API_KEY); Claude model; max response tokens
AUTH_ENABLED false Built-in login + RBAC (else front with SSO/proxy)
ADMIN_USER / ADMIN_PASSWORD admin / — Bootstrap admin on first run (random password logged if blank)
SESSION_TTL_HOURS / SESSION_COOKIE_SECURE 12 / false Session lifetime; set secure cookie over HTTPS
INGEST_QUEUE_MAX 10000 Async ingest queue capacity (live sources)
INGEST_WORKERS 2 Writer workers draining the queue
INGEST_FLUSH_MAX / INGEST_FLUSH_MS 2000 / 1000 Flush a buffer at N events or N ms
SYSLOG_ENABLED false Listen for syslog (UDP+TCP)
SYSLOG_UDP_PORT / SYSLOG_TCP_PORT 5514 / 5514 Syslog ports (0 disables a transport)
SYSLOG_FORMAT auto Fixed parser for messages, or auto (per-message detect)
SYSLOG_TLS_CERT / SYSLOG_TLS_KEY — Enable TLS on syslog-over-TCP

Project layout

SIEM-Lite/                  # repo root (product: LogOcean)
├── docker-compose.yml      # Postgres + app
├── Dockerfile
├── schema.sql              # partitioned events table, FTS, indexes, batches, cim_models + cim_meta
├── requirements.txt
├── app/
│   ├── main.py             # FastAPI routes + UI + lifespan
│   ├── api.py              # HTTP API: POST /api/v1/ingest + POST /api/v1/query (LOQL)
│   ├── loql/               # LOQL: nodes (AST) → parser → compiler (parameterized SQL) → run
│   ├── cim/                # CIM data models: spec (contract) + registry (models.yaml) +
│   │                       #   match (Python membership evaluator) + sql (cim_<tag> views)
│   ├── config.py
│   ├── db.py               # pool, partitions, insert, search, stats, purge, api_keys
│   ├── pipeline.py         # source-agnostic parse → normalize → insert core
│   ├── ingest.py           # per-batch orchestration (sha, batch, source tagging)
│   ├── streaming.py        # bounded async ingest queue + batching writer workers
│   ├── receivers/syslog.py # UDP/TCP/TLS syslog receiver → queue
│   ├── detection/          # engine.py (per-event Sigma-subset), correlation.py, runtime.py
│   ├── alert_actions.py    # fan new alerts to notifications + response
│   ├── notify/             # webhook + email channels, background dispatcher
│   ├── response/           # agentless playbooks + audit log + stateful auto-revert
│   ├── collectors/         # pull connectors (Okta/GitHub/GitLab/AWS/Entra/M365/GCP) + scheduler
│   ├── threatintel/        # IOC matcher + feed loader + index runtime
│   ├── triage/             # suppression/allowlist matcher + index runtime
│   ├── navigator.py        # ATT&CK Navigator layer export (pure)
│   ├── risk.py             # UEBA entity extraction + risk scoring (pure)
│   ├── killchain.py        # kill-chain / attack-story reconstruction (pure)
│   ├── killchain_runtime.py # DB-backed reconstruction + auto-create scheduler
│   ├── ot.py               # OT/ICS analytics: asset/conversation classification (pure)
│   ├── workbench.py        # detection workbench: rule tester + coverage + health (pure)
│   ├── coverage.py         # ATT&CK (enterprise+ICS) + ATLAS detection-coverage scoreboard
│   ├── sigma_import.py     # translate community SigmaHQ rules → our engine (logsource gate)
│   ├── copilot/            # AI SOC copilot — prompts.py (pure) + client.py (Claude SDK wrapper)
│   ├── saved.py            # saved-search path/query validation + target URL (pure)
│   ├── severity.py         # canonical severity order + roll-up
│   ├── detect.py           # format auto-detection
│   ├── normalize.py        # dedup hash + full-text blob
│   ├── models.py           # NormalizedEvent
│   ├── auth.py             # password hashing (pbkdf2), roles, RBAC dependency
│   ├── compliance.py       # MITRE technique → framework control mapping + report
│   ├── util.py             # tolerant time/IP/int coercion; API-key helpers
│   ├── parsers/            # 29 vendor/format parsers (+ zeek_ics OT/ICS enrichment helper)
│   ├── templates/          # server-rendered Jinja2 pages (+ _macros chart partials)
│   └── static/style.css
├── rules/                  # detection + correlation rules (Sigma-subset YAML) · imported/ (SigmaHQ imports)
├── playbooks/              # agentless response playbooks
├── clients/                # logocean_push.py (push helper) · logocean_import.py (bulk file import)
├── scripts/                # coverage_report.py (coverage + Navigator layers) · import_sigma.py (SigmaHQ import)
├── docs/                   # LOQL.md · CIM.md · SPLUNK_TRANSFORMATION_ROADMAP.md ·
│                           #   DETECTION_COVERAGE_ROADMAP.md (living detection-coverage plan)
├── samples/                # one example file per format (+ samples/sigma/ importer fixtures)
└── tests/                  # unit (DB-free) + integration (real Postgres) tiers

Testing & CI

The suite has two tiers. Unit tests are DB-free and run anywhere; integration tests (marked integration) exercise a real PostgreSQL and self-skip when DB_DSN is unset. Currently 885 unit + 67 integration (952 collected).

Note the asymmetry: an unset DB_DSN skips, but a DSN that is set and broken fails the run rather than skipping it, and a session that collects integration tests with a DSN configured and then executes none of them also fails. Both guards live in tests/conftest.py — see CLAUDE.md for why.

pip install pytest python-dateutil
# Unit only (no database) — the default local experience:
PYTHONPATH=. python -m pytest tests/ -m "not integration" -q   # PowerShell: $env:PYTHONPATH="."

# Integration (needs Postgres + httpx); point DB_DSN at a throwaway database:
pip install httpx
DB_DSN=postgresql://logocean:logocean@localhost:5432/logocean \
  PYTHONPATH=. python -m pytest tests/ -m integration -q

The unit tier covers parsers + auto-detection (over the bundled samples), API-key auth, the async ingest queue (grouping, worker loop, backpressure), syslog TCP framing, gzip ingest decompression (bomb-guarded) + the bulk-import client's line-aligned chunker, the detection engine (Sigma-subset matching, all field modifiers + condition grammar, incl. the Tripwire-FIM, Sysmon/endpoint, Nutanix, and OT packs), the SigmaHQ importer (translation, logsource gating, skip reasons), the coverage scoreboard + rule-linter, inline detection in the pipeline, correlation-rule loading/dedup, notification routing + dispatcher, response playbook matching/execution, collector URL/cursor logic (incl. AWS SigV4 + Microsoft OAuth + GCP signed-JWT helpers), threat-intel (IOC classification, feed parsing, matching + alerting), suppression/allowlist matching, case severity roll-up, the ATT&CK Navigator builder, UEBA entity extraction + risk scoring, kill-chain reconstruction, the detection workbench, the AI copilot (prompt construction + Sigma extraction against a fake client), auth (pbkdf2, role ranking, the RBAC dependency), the audit helper, the OT/ICS enrichment + rule pack, and the compliance report — all without a database. It also covers both transformation backbones as pure functions: LOQL (parser + emitted-SQL snapshots, the "every value is a bound parameter" invariant, and the datamodel source) and CIM (registry loader guards, byte-exact view DDL, evaluator equivalence between the compiled and reference membership walks, and a per-source tag expectation over every bundled sample — so a regression names the source it broke).

The integration tier runs against an actual PostgreSQL 16 and verifies what mocks can't: month-partition auto-creation, the GIN full-text index, inet/CIDR search, ON CONFLICT dedup, retention purge dropping whole partitions, the correlation SQL, the pipeline raising and suppressing alert rows, alert/case round-trips, the alert analytics aggregations, UEBA baselines / anomalies / risk ranking, kill-chain story→case creation, the workbench windowed stats, the coverage endpoints, and the HTTP stack end-to-end (TestClient → API-key auth → ingest → detect, plus the dashboard / report / Navigator / CSV endpoints). CI runs the unit tier on Python 3.11–3.13 and the integration tier against a Postgres service container (.github/workflows/tests.yml).

Data model & retention

  • events is partitioned BY RANGE (event_time); partitions are monthly (events_YYYYMM) and created on demand at ingest. A events_default partition catches out-of-range timestamps. Time-range searches prune to the relevant months.
  • Retention = dropping whole monthly partitions older than the cutoff (instant, no row-by-row delete). The Admin page exposes a guarded purge; the floor is RETENTION_YEARS. For 3-year retention you typically never purge — set AUTO_PURGE=true only when you want to stop keeping data beyond the floor.
  • Idempotent ingest — every record has a dedup hash, so re-uploading the same file (or overlapping exports) does not create duplicates.

    One-time migration note (Windows Security only). windows_security.py now writes the resolved Event ID back into the record as raw["event_id"], which is what makes the CIM Authentication / Endpoint / Change clauses reachable for Windows. The dedup hash is taken over raw, so every Windows Security event's identity changed: re-uploading a previously-ingested Windows Security export will now insert duplicates instead of deduping. Delete the old batch first, or accept them. No other parser's raw shape changed, and dedup works normally from here on.

  • CIM membership — each row also carries cim_models text[] (GIN-indexed), the tags of the CIM data models it belongs to, stamped in Python at ingest. A cim_meta row records which registry the views and the stored tags were derived from, so the app can tell you when a membership backfill is due.
  • Timezone — events are stored in UTC (partitioning, dedup, retention and cross-source correlation all depend on it) and displayed in IST (UTC+05:30) in the UI and CSV exports. The conversion is render-time only — a fixed offset with no daylight saving, so it needs no system time-zone database.
  • Scale — tuned for manual-upload volumes (tens of millions of rows). For very high ingest, batch larger files, add a read replica, or move hot search to OpenSearch.

Parser accuracy notes

  • Palo Alto CSV maps by column header (robust across PAN-OS versions).
  • Palo Alto syslog uses documented positional field maps for Traffic/Threat/ System/Config (PAN-OS 10/11 common layout). The complete positional field list is preserved in raw.fields, so even if a field index drifts on your PAN-OS version, the data is retained and searchable, and the maps in app/parsers/paloalto_syslog.py are easy to adjust.
  • CrowdStrike CSV/JSON resolve each field from multiple candidate names to cope with detection vs incident vs FDR shapes.
  • Cisco ASA/Firepower mines the 5-tuple, bytes and user from the free-text message (best-effort src/dst, from/to, Built for/to); the full message is in raw.
  • Zeek reads the #separator / #fields / #path header, so column order is taken from the file itself; a file may concatenate several logs (each with its own header).
  • Cloud/identity JSON (CloudTrail, GCP, Azure, M365, Entra, Okta, GitHub, GitLab) is routed by record keys and resolves fields case-insensitively to tolerate camelCase (Graph) vs PascalCase (Azure Monitor) and wrapper shapes ({"Records":…}, {"records":…}, {"entries":…}, {"value":…}).
  • Generic JSON is the JSON catch-all: it flattens one level so Elastic Common Schema keys (source.ip, event.action, user.name) resolve, and maps a wide set of candidate field names; anything unmapped stays in raw. It is the JSON fallback, so a recognized source is never shadowed by it.

Security

Set AUTH_ENABLED=true for built-in login + RBAC (roles: admin / analyst / viewer). An admin is bootstrapped on first run from ADMIN_USER/ADMIN_PASSWORD (a random password is logged if blank); manage users from the Admin page. Passwords are pbkdf2-hashed and sessions are server-side (revocable). Use SESSION_COOKIE_SECURE=true behind HTTPS. Every response carries security headers (X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy, and a CSP with frame-ancestors 'none'); with auth on, state-changing UI requests also get a CSRF same-origin (Origin/Referer) check, and the last enabled admin can't be demoted or disabled (no self-lockout). Security-relevant actions (login/logout, purge, key/rule/collector/user changes, alert triage, upload) are recorded in an audit log on the Admin page. With auth off, run behind your SSO/reverse proxy or on a trusted network. Either way, keep the Postgres volume backed up (it is your 3-year archive).

Input hardening. Ingest treats all log content as untrusted: uploads and the API body are size-capped (MAX_UPLOAD_MB) and streamed so a huge payload can't exhaust memory; a deeply-nested JSON "bomb" is rejected before parsing by an explicit depth guard (_MAX_JSON_DEPTH, version-stable — not reliant on the interpreter raising RecursionError); CSV exports neutralize spreadsheet formula injection; and correlation/search SQL uses whitelisted columns with fully parameterized values.

Roadmap

A separate, longer-range plan grows LogOcean toward Splunk-class analytics — see docs/SPLUNK_TRANSFORMATION_ROADMAP.md. Its two Phase-1 backbones have shipped: LOQL (piped query language → parameterized SQL) and CIM data models (datamodels: rules, from datamodel: searches, 11 models). Still open in Phase 1: window verbs (eventstats/streamstats/transaction), rex/spath + macros, a durable ingest queue with ack, and a published EPS / bytes-per-event benchmark.

Detection coverage is grown as its own phased programme — see docs/DETECTION_COVERAGE_ROADMAP.md for the living plan. Highlights: a measured ATT&CK (Enterprise + ICS) + ATLAS coverage scoreboard (done), a community SigmaHQ importer (done), curated high-fidelity endpoint / cloud / identity packs, a behavioural upgrade (temporal-sequence + distinct-count correlation, GeoIP enrichment for impossible-travel), and an adversarial-AI (ATLAS) track built on an LLM / AI-gateway telemetry parser.

License & attribution

Author: Krishnendu De · Co-author: Claude Fable 5.0.

See LICENSE for terms. Imported SigmaHQ rules retain their original authorship and are used under the Detection Rule License (DRL); MITRE ATT&CK and ATLAS are trademarks of The MITRE Corporation.

Secrets vault

Collector credentials encrypted at rest with AES-256-GCM, so a database dump, backup or read replica carries ciphertext only. The master key lives in VAULT_KEY and not in the database it protects.

pip install cryptography          # optional dep: the stdlib has no AES
python -c "from app.vault.crypto import generate_key; print(generate_key())"
VAULT_ENABLED=true
VAULT_KEY=<base64, 32 bytes>

Resolution is vault first, then the environment variable, so integrations migrate one at a time instead of all at once on restart. Manage secrets on Admin ▸ Secrets vault: store a credential, import the existing plaintext environment variables, or rotate the master key (all-or-nothing — every secret is re-sealed in one transaction or none is).

Full reference: docs/VAULT.md.

About

An Open-source Security Information and Event Management (SIEM) solution blended with UEBA and TI Capabilities

Topics

awsazurecefcrowdstrike-falcongcpiec101iec104iec62443leefmodbusng-siempaloaltonetworksplc-controllerqradarrtu-over-tcpsecurity-operation-centersentinelsiem-developmentsoarsplunk

Resources

Readme
MIT license
Activity

Stars

3 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages