🕵️ WireWarden – Hybrid Python/C NIDS & Observability Engine

A full end-to-end real-time Network Intrusion Detection & Prevention System (NIDS/NIPS).
Combines a multi-threaded Python (Scapy) capture and analysis engine with a dual-layer DPI pipeline — an Aho-Corasick automaton for cleartext-credential patterns and a native C library (bounds-checked, called live via ctypes) for injection-pattern signatures — plus sliding-window anomaly detection, time-limited OS-firewall Active Defense, and a fully code-managed monitoring stack (Dashboard as Code) on Docker (Grafana + Loki + Promtail).

Overview · Dashboard · Features · DPI · Resilience · Quick Start · Testing · Audit · Limitations · Structure

---

Real engine output on simulated attack traffic (localhost, dry-run).

---

✨ At a Glance

Detects	SYN floods, port scans, stealth scans (NULL / FIN / XMAS), DNS tunneling, injection attempts (SQLi, XSS, path traversal, command injection) and cleartext credentials
Responds	Time-limited iptables / netsh blocks, removed on expiry and on shutdown, with a never-block list and a --dry-run mode
DPI engine	Memory-safe C scanner at ≈ 1M packets/s per call (≈5x a naive Python loop; on par with Aho-Corasick on small payloads and up to ≈1.5x faster on larger ones). Profiling shows Scapy dissection, not DPI, dominates per-packet time: see the performance profile
Quality	57 pytest tests, C engine fuzzed under AddressSanitizer + UBSan, CI on every push
Documented trade-offs	Known limitations and the bugs found by auditing my own code are written up with their fixes

---

🔎 Overview & Architecture

WireWarden monitors, analyzes, and responds to network security events across OSI layers 3, 4, and 7. The architecture is a continuous Producer-Consumer pipeline that separates low-level packet capture, real-time threat analysis, and structured observability shipping.

flowchart LR
    NIC[📡 NIC] -->|Raw Packets| SnifferThread[🐍 Sniffer Thread<br>Scapy store=0]

    subgraph Engine [Python NIDS Core Engine]
        SnifferThread -->|Non-blocking put by flow| Queue[📦 Per-flow queues<br>one per worker, 20,000 total]
        SnifferThread -. queue full .-> Drops[📉 dropped counter]
        Queue -->|Same flow, same worker| Worker[⚙️ Worker Pool]

        subgraph Detection [Detection & Defense]
            Worker -->|Inbound only| Anomaly[🛡️ DoS / Port-Scan<br>Sliding Windows]
            Worker -->|Payload content| DPI[⚡ Aho-Corasick + Native C DPI<br>+ per-flow tail]
            Worker -->|Queries only| DNS[🧬 DNS Entropy]
            Anomaly -->|Threshold + not whitelisted| Firewall[🧱 FirewallManager<br>iptables / netsh]
        end

        GC[🧹 GC Thread] -->|Purge windows| Anomaly
        GC -->|Expire → remove rule| Firewall
    end

    Worker -->|JSON Log| LogFile[📄 logs/network_security.json]
    LogFile --> Promtail[🔄 Promtail Container]
    Promtail -->|Push| Loki[🗄️ Loki DB]
    Loki --> Grafana[📊 Grafana Dashboard]

Loading

---

📊 Dashboard Preview

Live view of the WireWarden Security Overview Grafana dashboard, provisioned automatically from code.

Live JSON log stream with DNS query events, alongside real-time Security Events Distribution and Threat Timeline panels.

Active Defense in action — a DoS/SYN Flood attack triggers automatic IP isolation, alongside a DNS Tunneling detection alert (Shannon entropy-based).

Full event-type breakdown (Port Scans, Stealth Scans, DPI Alerts, DoS Attacks) with Top Suspicious Source IPs and Total Security Alerts panels.

---

🚀 Core Features

Domain	Feature	Description	Performance Indicator
📡 Network	Real-time L3-L7 Sniffing	Real-time capture and analysis of IP, TCP, UDP, and DNS traffic while keeping memory bounded (store=0, bounded queue).	Bounded memory, O(1) enqueue, counted drops
🛡️ Cyber Security	Sliding-Window & Stealth Detection	Detects DoS (SYN Flood — pure SYN only, replies excluded), port scans (per (protocol, port); UDP answers to traffic this host started are not probes), and Stealth Scans (NULL, FIN, XMAS). DoS/scan windows count inbound traffic only.	O(1) window updates
🧬 DNS Security	DNS Tunneling Detection	Shannon entropy and length of the leftmost label, on queries only (responses are skipped so the resolver is never blamed).	O(L) per label
⚡ Active Defense	Time-limited IP Isolation & OS Firewall	Adds netsh / iptables rules on a DoS verdict, removes them on expiry and shutdown (a failed removal is retried), never blocks whitelisted/local/gateway addresses, supports --dry-run, and logs the real command result.	Non-blocking (background thread), O(1) blacklist check
🔍 DPI Engine	Deep Packet Inspection	Dual-layer L7 scanning: Aho-Corasick for cleartext-credential patterns, plus a native C library (ctypes) for 14 case-insensitive injection signatures. URL-decoding and cross-segment matching included. Falls back to Aho-Corasick alone if the C library isn't built.	O(N+M) keyword match; C path ≈1M pkt/s per call (≈5x a naive Python loop, up to ≈1.5x Aho-Corasick on larger payloads)
📊 Observability & IaC	Dashboard as Code (Grafana + Loki)	A single all-in-one "WireWarden Security Overview" dashboard in standard JSON, auto-loaded on container startup via provisioning files.	Instant provisioning on boot
📝 Logging	Structured JSON Dual-Stream	Colorized console output alongside structured JSON log writes (logs/network_security.json), tailored for Promtail.	Promtail-ready labels: level, event_type, src_ip

🧬 Algorithmic Detection: Shannon Entropy for DNS Tunneling

WireWarden identifies covert communication and data exfiltration over DNS by measuring the randomness (entropy) of the leftmost label of each DNS query.

Ordinary domain names follow predictable natural-language patterns, whereas encoded data streams (e.g. Base64/Hex DNS tunneling) produce much higher entropy.

• Mathematical Model: Shannon Entropy $H(X)$ over the character distribution of the label: $$H(X) = -\sum_{i=1}^{n} P(x_i) \log_2 P(x_i)$$
• Detection Rule: a query is flagged when the label's $H(X) > 4.3$ or its length exceeds 45 characters.
• Why both conditions: entropy of an $L$-character label cannot exceed $\log_2 L$ bits, so $H > 4.3$ is only reachable for labels of about 20+ characters. Short labels can't trigger it, which keeps ordinary hostnames quiet; the length rule catches very long labels regardless of their character mix.

---

🔬 DPI Engines: Live Detection vs. Throughput Benchmark

WireWarden runs two DPI engines together in the live pipeline (main.py), each covering a different signature class, plus a standalone script that measures the C engine's throughput in isolation:

	Live: Aho-Corasick (KeywordEngine)	Live: Native C (NativeSignatureEngine)
Technology	pyahocorasick automaton, substring fallback if unavailable	Native C via ctypes, loaded once in NetworkGuardian.__init__
Signature set	password=, passwd=, pwd=, authorization: basic, api_key=, apikey=, select * from	14 case-insensitive signatures: SQLi, XSS, path traversal, command injection, sensitive files (defined once in c_src/dpi.c, read back by Python)
Purpose	Cleartext credential / query leak alerting	Injection-pattern alerting
Required?	Recommended (pip install pyahocorasick)	Optional — compile with make -C c_src; without it WireWarden runs on keyword DPI only

Both engines apply to the same normalized payload: URL-decoded (%2e%2e%2f → ../), and joined with a short per-flow tail so a signature split across two consecutive TCP segments is still caught.

---

🏎️ DPI Engine Performance & Profile

• Single-pass, bounds-checked scan — A 256-entry first-byte table rejects most bytes with one lookup; only candidate positions are compared. A signature is compared only if it fits inside the buffer, so no read can pass payload_len - 1 — safe on raw, non-NUL-terminated binary traffic. Verified by fuzzing under ASan/UBSan.
• Benchmark — python benchmark_dpi.py asserts all engines agree, then times the same 14 signatures (all case-insensitive) on 100,000 identical short payloads:

Engine	Time	Throughput	vs. naive
Python loop (lowercase + sig in payload)	0.439 s	227,769 pkt/s	1.00x
Python regex (one compiled alternation, IGNORECASE)	0.523 s	191,073 pkt/s	0.84x
Aho-Corasick (pyahocorasick)	0.104 s	964,561 pkt/s	4.23x
C, per-call via ctypes (what the live pipeline does)	0.093 s	1,074,722 pkt/s	4.72x
C, batch (one FFI call for all packets)	0.024 s	4,117,039 pkt/s	18.08x

Representative run on Windows, Python 3.13. Across ten runs the C engine was stable (0.09–0.11 s per call, 0.024–0.027 s in batch) while the pure-Python baseline varied (0.44–0.68 s), so the speedup over the naive loop ranged ≈4.5–7.3x per call (median ≈5.1x) and ≈17.8–27.4x in batch (median ≈19.6x). Re-run benchmark_dpi.py on your machine.

Versus Aho-Corasick, on varied payloads

A naive loop is the weakest Python baseline. The project's own alternative is pyahocorasick, so python benchmark_dpi.py --sweep compares against it on varied HTTP-like payloads from 64 to 1460 bytes (5% malicious), in packets per second:

Payload	Python loop	Python regex	Aho-Corasick	C per call	C batch	C per call vs. best Python
64 B	191,752	231,568	988,303	945,533	2,278,086	0.96x
256 B	142,598	53,744	324,261	430,695	610,188	1.33x
1024 B	73,570	14,314	95,924	141,092	155,877	1.47x
1460 B	56,649	10,327	68,026	106,626	114,994	1.57x

Windows, Python 3.13, best of 5. The C engine is on par with pyahocorasick on small payloads and 1.3–1.6x faster from 256 bytes up. A Linux run (Python 3.12) showed parity at all sizes (0.8–1.1x), so treat the gap as platform-dependent. Besides throughput, the C engine is a memory-safe, single-pass, case-insensitive scanner with no extra dependency.

Where per-packet time goes

python benchmark_pipeline.py measures one thread on synthetic HTTP-over-TCP frames (about 100-byte payloads):

Step	µs / packet	Share
Scapy dissects raw bytes into layers	207	≈80%
analyze_packet: all detectors + DPI	52	≈20%
of which both DPI engines (+ URL decode, flow tail)	3.5	≈1.4%
of which the C signature scan alone	1.1	≈0.4%

Windows, Python 3.13. A Linux run (Python 3.12) gave the same picture: dissection 78–82%, all DPI about 2%. End to end this is ≈3,900 packets/s per capture thread. Scapy's dissection dominates per-packet time and the whole DPI stage is only a few percent, so a faster DPI engine cannot raise end-to-end throughput; the next optimization target is the packet parser. Above this rate the bounded queue drops packets, and every drop is counted.

How to read the numbers: the batch row is the C engine's ceiling with call overhead amortized. The live pipeline calls C once per packet, so the per-call rows are the representative figures for main.py. None of the micro-benchmarks is end-to-end throughput; benchmark_pipeline.py is.

---

⚡ Performance & Resilience Analysis

• Memory Backpressure & Counted Drops — Bounded queues (20,000 packets in total, split across the workers) combined with store=0 in Scapy keep memory bounded: the sniffer never accumulates packets. When analysis falls behind, excess packets are dropped rather than risking Out-Of-Memory crashes — and every drop is counted and reported ([STATS] captured=… dropped=…), so loss is visible, not silent.
• Concurrency Model — Workers are threads, and under CPython's GIL pure-Python analysis is serialized. Threads decouple capture from analysis and overlap work that releases the GIL (socket reads, the ctypes call into C). Scaling analysis across cores is a multiprocessing step.
• Per-Flow Worker Affinity — Each worker has its own queue and a flow (both directions) always goes to the same one, so a worker sees a flow's segments in capture order. With a single shared queue, 2–8 workers missed 0.2–0.8% of signatures split across two segments; with per-flow queues all of them are found at any worker count.
• Thread Safety via a Shared Lock — Shared state (syn_history, port_history, blacklist) is protected by a single threading.Lock for atomic updates. All time windows use time.monotonic(), so wall-clock changes (NTP, DST) cannot corrupt them.
• Active Defense & OS Firewall Integration — Dynamic OS firewall rules (iptables on Linux, netsh advfirewall on Windows) are applied on a DoS verdict in a background thread, so slow firewall commands don't stall the worker. Blocks are time-limited: the rule is removed when the block expires and on shutdown, and the log reports what the OS actually returned (applied / FAILED + stderr). A rule stays tracked until the OS confirms its removal: failed removals are retried every cleanup cycle, and anything still left at shutdown is reported with the exact command to run by hand. A never-block list (loopback, this host, default gateway, --whitelist) and a --dry-run mode guard against self-inflicted outages.
• Safe Defaults for the Observability Stack — Loki and Grafana are bound to 127.0.0.1, the Grafana image version is pinned, and the admin password is mandatory (docker compose refuses to start without it).
• Deterministic Resource Cleanup (Garbage Collector) — Dormant IP records, expired blacklist entries and stale flow tails are purged every 30 seconds by a background thread, keeping memory steady under sustained traffic.

---

📋 Prerequisites

• Git — Required to clone the repository.
• Docker & Docker Compose — For running Loki, Promtail, and Grafana.
• Python 3.10+ — Required for the NIDS engine and test suite.
• GCC / Make — Required to compile the native C DPI library (libdpi.so on Linux, libdpi.dylib on macOS, libdpi.dll on Windows). Recommended before running main.py; also needed for benchmark_dpi.py and the tests. Without it WireWarden still runs on keyword DPI alone.
• pyahocorasick — Recommended for the keyword engine; a substring fallback is used automatically if it's missing.
• Administrator / Root Privileges — Required to capture raw sockets via Scapy and to inject OS firewall rules (or use the least-privilege setcap option below on Linux for capture).
• tcpdump or libpcap (Linux) — Scapy needs it to compile the ip capture filter. Without it sniff() fails on startup.
• Npcap (Windows only) — Required for Scapy to capture raw packets on Windows adapters.

---

⚙️ Installation & Quick Start

# 1. Clone the repository
git clone https://github.com/RazEini/WireWarden.git
cd WireWarden

# 2. Environment Setup
cp .env.example .env  # Set GRAFANA_ADMIN_PASSWORD — docker compose refuses to start without it

# 3. Start Observability Stack (Grafana, Loki, Promtail)
# Grafana automatically provisions all dashboards from grafana/dashboards/
docker compose up -d

# 4. Setup Python Environment
python -m venv .venv
.\.venv\Scripts\activate     # On Windows
source .venv/bin/activate    # On Linux/Mac
pip install -r requirements.txt

# 5. Compile the Native C DPI Library (recommended)
# main.py auto-detects and loads it at startup; without it WireWarden runs on keyword DPI only.
make -C c_src                                          # Linux / macOS
gcc -shared -O3 -o libdpi.dll c_src/dpi.c              # Windows — requires MinGW/MSYS2

# Optional: run the standalone benchmark
python benchmark_dpi.py

6a. Run NIDS Engine — Linux / macOS

# Safe first run: full detection, but never touches the firewall
sudo .venv/bin/python main.py --dry-run

# Principle of Least Privilege - grant raw-socket capability without full sudo
# (firewall rules still need root, so combine with --dry-run if not running as root):
sudo setcap cap_net_raw,cap_net_admin=eip $(readlink -f .venv/bin/python)
.venv/bin/python main.py --dry-run

# Real run with active defense and a never-block range:
sudo .venv/bin/python main.py --whitelist 10.0.0.0/24

6b. Run NIDS Engine — Windows

# Run PowerShell / CMD as Administrator:
python main.py --dry-run
python main.py --whitelist 10.0.0.0/24

Flag	Default	Meaning
--dry-run	off	Log what would be blocked; never run firewall commands
--whitelist A,B/CIDR	–	Extra never-block addresses (this host, gateway and loopback are automatic)
--include-outbound	off	Lab mode: also count traffic this host sends. Needed when test_attack.py targets another device such as your router (see Running the simulator)
--block-seconds	300	How long a DoS source stays blocked
--dos-threshold / --scan-threshold	500 / 50	Events per 10-second window
--workers / --iface	4 / default	Worker threads / interface to sniff

7. Run Attack Simulator (in a separate terminal)

# Target a specific address (see "Running the simulator" below for which one to choose):
python test_attack.py <TARGET_IP>

📊 Accessing Grafana: open http://localhost:3000 (username: admin, password from .env). All dashboards load automatically.

Which traffic counts? (inbound vs. outbound)

DoS and port-scan windows count inbound traffic only, meaning packets whose destination is one of this machine's own addresses. Your own outbound traffic and replies must never look like an attack. DPI, stealth-scan and DNS-tunneling detection look at traffic in both directions.

Running the simulator

The target you choose decides which alerts you see, because the engine only sees packets that cross the interface it captures on.

Setup	Commands	What you see
Everything on this machine (Linux)	python main.py --iface lo --dry-run python test_attack.py 127.0.0.1	All detectors, including the DoS block. Loopback traffic is inbound by definition.
Another device on your network (router, second PC; used on Windows)	python main.py --include-outbound --dry-run python test_attack.py <DEVICE_IP>	All detectors. The packets leave your machine, so the lab flag is needed for the DoS and port-scan alerts.
Another device, without the lab flag	python main.py --dry-run python test_attack.py <DEVICE_IP>	DPI, stealth-scan and DNS alerts only. The DoS and port-scan windows ignore outbound traffic on purpose.

--dry-run keeps every run alert-only, with no firewall rule. Traffic that arrives from another computer is genuinely inbound, so a simulator run from a second machine against this host's address needs no lab flag.

Running python test_attack.py without an argument targets this machine's own address. Packets a machine sends to itself usually never reach the capture interface, so the simulator prints a reminder to use one of the setups above.

---

📝 JSON Log Structure (Structured Logging)

{
  "timestamp": "2026-10-20T10:30:15.123456+03:00",
  "level": "WARNING",
  "message": "[PORT SCAN DETECTED] Host 203.0.113.7 probed 61 unique ports",
  "logger": "NetworkGuardian",
  "src_ip": "203.0.113.7",
  "event_type": "PORT_SCAN",
  "details": "61 ports scanned"
}

Event types: PORT_SCAN, STEALTH_SCAN, DOS_ATTACK, DOS_ATTACK_WHITELISTED, DNS_TUNNELING, DNS_QUERY, DPI_ALERT, DPI_ALERT_NATIVE, FIREWALL_BLOCK, FIREWALL_UNBLOCK, FIREWALL_ERROR, FIREWALL_DRY_RUN, ENGINE_STATS.

---

🧪 Testing & Quality Gates

pip install -r requirements-dev.txt
make -C c_src
python -m pytest tests -v        # 57 tests: detection logic, DPI engines, firewall manager (iptables + netsh), lab mode
make -C c_src fuzz ITER=1000000  # C engine under AddressSanitizer + UndefinedBehaviorSanitizer

• Regression tests pin every defect found during the project's own audit (see the table below). Each test was verified to fail when its bug is reintroduced.
• Firewall tests verify the exact iptables and netsh commands that get built, that add and delete use the same rule name, and that a failed command is never recorded as applied. They use an injected command runner, so they run on any OS without admin rights; the netsh commands were also run manually on Windows.
• Fuzzing allocates every buffer at its exact size, so an out-of-bounds read is an ASan failure. It covers truncated and upper-cased signatures placed flush against the end of the buffer.
• CI (.github/workflows/ci.yml) builds the C library with -Wall -Wextra -Werror, runs pytest on Python 3.10 and 3.12, and runs the ASan/UBSan fuzzer.
• test_attack.py is a live simulator (synthetic traffic only) that exercises the real sniff path and the Grafana dashboards, including a SYN-ACK negative control that must not alert.

🐞 Bugs Found by Auditing My Own Code

After the first working version I audited it, reproduced each problem with a test, and fixed it. Every row has a regression test. The last three rows came from a later independent review; I reproduced each one before fixing it.

Defect	Impact	Fix
SYN-ACK counted as SYN	A busy server answering clients was blocked as a "DoS" source	Count only SYN without ACK
Outbound traffic counted	The host's own connections and replies looked like attacks	DoS and port-scan windows count inbound only
DNS responses inspected	The alert blamed the resolver instead of the asking host	Inspect queries only
Firewall rules never removed	The log said "5 minutes" but the rule lasted forever	Track rules; remove on expiry and on shutdown
"Rule applied" logged before the command ran	A failed iptables / netsh looked like success	Log the real exit status and stderr
DPI skipped ports like 443	Plaintext attacks on those ports slipped through	Choose payloads by content, not by port
Case-sensitive signatures	union select in lowercase evaded detection	ASCII case-insensitive matching in C
No handling of split segments	A signature cut across two packets evaded detection	Per-flow tail carried between segments
Log handlers added on every engine start	Duplicate log lines	Add handlers only once
Firewall rule forgotten before the OS confirmed removal	A failed iptables / netsh delete left a permanent block that nothing tracked or retried	Keep the rule tracked until removal succeeds; retry every cycle; report leftovers with the manual command
Every UDP packet counted as a scan probe	60 legitimate DNS answers raised a PORT_SCAN alert against the resolver	Answers to traffic this host started are replies, not probes
One shared queue for all workers	Two workers could process a flow's segments in swapped order and miss a split signature (0.2–0.8% with 2–8 workers)	Per-flow queues: a flow always goes to the same worker

---

⚠️ Known Limitations

WireWarden is built for labs, home and small networks, and is not meant to replace production IDS platforms such as Suricata or Zeek. Current scope:

• Not a full TCP reassembler. A short per-flow tail catches signatures split across consecutive, in-order segments (per-flow worker affinity keeps them in capture order). Segments that arrive out of order on the wire, retransmissions and heavy fragmentation are not reassembled.
• Signature-based. 14 literal patterns catch common, unobfuscated payloads. Comment-based SQL (UNION/**/SELECT), double encoding and similar obfuscation are not covered.
• IP spoofing. SYN floods use forged sources by nature, so blocking the "source" can block an innocent address. Mitigations: whitelist critical infrastructure, time-limited blocks, --dry-run.
• IPv4 only, and no TLS inspection (encrypted payloads are skipped by design).
• Throughput is bound by packet dissection. Scapy parsing costs ≈150–200 µs per packet in the profile above (≈4–6K packets/s per capture thread), which suits lab, home and small-office traffic. Above that rate packets are dropped and counted; high-rate links would need a lighter parser.
• Static thresholds (--dos-threshold, --scan-threshold) need tuning per network.
• Direction needs local addresses. If none can be discovered, all traffic is treated as inbound.

---

📂 Project Structure

WireWarden/
├── assets/                          # Static Documentation Assets (Dashboard Screenshots)
│   ├── dashboard_overview.png
│   ├── dashboard_threat_detection.png
│   └── dashboard_analytics.png
├── c_src/                           # Low-Level Native C Extensions
│   ├── dpi.c                        # Native C DPI Engine (single-pass, case-insensitive, bounds-checked)
│   ├── fuzz_dpi.c                   # ASan/UBSan fuzz harness
│   └── Makefile                     # Build setup + `make fuzz`
├── grafana/
│   ├── dashboards/                  # Standard JSON Dashboard (Git Version-Controlled)
│   │   └── dashboard-WireWarden Security Overview.json
│   └── provisioning/                # Grafana Automated Provisioning Configs
│       ├── dashboards/dashboards.yml
│       └── datasources/datasources.yml
├── tests/                           # pytest: detection logic, native engine, firewall manager
│   ├── conftest.py
│   ├── test_detection.py
│   └── test_dpi_native.py
├── .github/workflows/ci.yml         # Build (-Werror) + pytest + ASan/UBSan fuzz
├── logs/                            # Runtime Log Directory (Ignored by Git, not tracked)
├── .env.example
├── .gitignore
├── LICENSE
├── README.md
├── benchmark_dpi.py                 # DPI micro-benchmark (5 engines, --sweep for payload sizes)
├── benchmark_pipeline.py            # Per-packet cost breakdown (Scapy vs detectors vs DPI)
├── docker-compose.yml
├── main.py                          # NIDS Core Engine (NetworkGuardian, FirewallManager, DPI engines)
├── promtail-config.yml
├── pytest.ini                       # Keeps the live simulator out of pytest
├── requirements.txt / requirements-dev.txt
└── test_attack.py                   # Live Traffic Simulator

---

📄 License

This project is distributed under the MIT license – free to use and modify for educational and research purposes.

---

👨‍💻 Raz Eini (2026)