splunk-siem-lab

A hands-on Security Information and Event Management (SIEM) detection lab. It generates realistic synthetic security logs, ingests them into Splunk, and applies SPL (Search Processing Language) detections that identify four embedded attack scenarios: SSH brute force, web directory traversal, network port scanning, and a data-exfiltration beacon. Every detection is mapped to the MITRE ATT&CK framework.

The lab is self-contained and reproducible: because the attacks are planted at known locations in generated data, each detection can be verified to fire correctly, and the synthetic dataset can be regenerated at any time.

---

Table of contents

1. Background: what a SIEM is and why detection matters
2. Core concepts
3. Repository contents
4. The four attack scenarios
5. Procedure: running the lab end to end
6. How the detections work
7. Validation results
8. Design notes and limitations
9. Requirements
10. Possible extensions

---

1. Background

What a SIEM is

A SIEM (Security Information and Event Management) system centralizes log data from many different sources — servers, applications, firewalls, endpoints — so that it can be searched, correlated, and alerted on from one place. The value of a SIEM is not in any single log, but in the ability to ask questions that span sources: for example, did the same IP address that scanned a set of ports also attempt to log in? No individual log answers that; the correlation does.

Splunk is one of the most widely deployed SIEM platforms. The detection concepts demonstrated here — searching, field extraction, aggregation, correlation, and thresholding — transfer directly to other platforms such as Elastic/Kibana and Microsoft Sentinel, which differ in query syntax but share the same underlying model.

Why detection matters

Most intrusion activity leaves traces in logs before, during, and after a compromise: a burst of failed logins, a request for a file that should never be requested, a scan across many ports, or a steady trickle of data leaving the network. Detection engineering is the practice of turning those traces into reliable, reviewable rules that surface real attacks while staying quiet on the large volume of benign activity a normal environment produces. Keeping that signal-to-noise ratio high is the central challenge — a detection that fires on ordinary behaviour is worse than no detection at all, because it trains analysts to ignore alerts.

---

2. Core concepts

These concepts underpin every detection in the lab. Reading this section first makes the SPL searches in step 6 straightforward to follow.

SPL is a pipeline

Search Processing Language works like Unix pipes. A search begins with terms that select events, then passes results through a chain of transforming commands joined by the pipe character |. For example:

... | stats count by src_ip | where count > 20

reads as: retrieve the matching events, count how many occurred per source IP, then keep only the sources with more than twenty. Each stage receives the output of the previous one. Once this pipeline model is internalized, most SPL becomes readable at a glance.

Field extraction: from raw text to structured data

Raw logs are unstructured text, but detections operate on fields (named values such as src_ip or dst_port). Splunk extracts fields in two ways:

• Automatic extraction for well-known formats — for example the Apache web access log (access_combined) and key=value formatted logs, where Splunk parses src_ip=..., dst_port=... into fields without configuration.
• Search-time extraction for free-text formats, using the rex command and a regular expression — for example pulling the source IP out of an SSH log line such as Failed password for root from 203.0.113.66.

The distinction between search-time extraction (applied when a query runs) and index-time extraction (applied when data is ingested) is a foundational Splunk concept: this lab relies on search-time extraction, which is more flexible and does not require reprocessing stored data.

Signature-based versus behavioural detection

Two complementary detection philosophies appear throughout this lab:

• Signature-based detection matches a known-bad indicator — for example a scanning tool's user-agent string. It is precise and cheap, but blind to anything not already known.
• Behavioural / anomaly-based detection flags unusual patterns rather than known strings — for example an abnormal volume of failed logins, or the suspicious regularity of a network beacon. It can catch novel activity but carries a higher false-positive risk.

Mature detection programs layer both. This lab deliberately includes examples of each so the tradeoff is concrete rather than abstract.

Correlation over single events

The strongest detections do not stop at a single event. Detecting a burst of failed logins is useful; checking whether the same source subsequently succeeded is what distinguishes an attempted intrusion from a successful one. Designing detections around that follow-up question — the one an analyst would ask next — is central to effective detection engineering, and appears explicitly in the brute force detection below.

Mapping to MITRE ATT&CK

MITRE ATT&CK is a public, standardized catalogue of adversary techniques, each with a stable identifier (for example T1110 for brute force). Mapping detections to ATT&CK technique IDs allows a team to reason about coverage — which attacker techniques are actually observable — instead of merely counting alerts. Every detection in this lab cites the technique it addresses.

---

3. Repository contents

splunk-siem-lab/
├── generate_logs.py          # Generates synthetic logs with embedded attacks
├── logs/                     # Output: auth.log, access.log, firewall.log
├── detections/
│   └── SPL_DETECTIONS.md     # All SPL searches, explained and ATT&CK-mapped
├── README.md
└── .gitignore

detections/SPL_DETECTIONS.md is the detection reference for this lab. For each scenario it records the SPL query, the logic in plain language, and the MITRE ATT&CK mapping. The relationship is one-to-one: every attack embedded by generate_logs.py has a corresponding detection documented there. This README explains the why and the how to run; SPL_DETECTIONS.md is the catalogue of the searches themselves.

---

4. The four attack scenarios

generate_logs.py produces three log sources containing normal background activity plus four deliberately embedded attacks. Because their locations are known, each detection can be verified against a definite ground truth.

#	Attack	Log source	MITRE ATT&CK
1	SSH brute force — many failed logins followed by one success	auth.log	T1110 (Brute Force), T1078 (Valid Accounts)
2	Web directory traversal and scanner activity	access.log	T1190 (Exploit Public-Facing App), T1595 (Active Scanning)
3a	Port scan — one source touching many ports on one host	firewall.log	T1046 (Network Service Discovery)
3b	Data-exfiltration beacon — large regular outbound transfers	firewall.log	T1041 (Exfiltration Over C2), T1071 (Application Layer Protocol)

All addresses used for the simulated attackers are drawn from the RFC 5737 documentation ranges (203.0.113.0/24, 198.51.100.0/24), which are reserved for examples and are non-routable — the data is unambiguously synthetic and safe.

---

5. Procedure

The lab runs in a Linux environment (including WSL on Windows). The steps proceed from generating data, to installing Splunk, to ingesting the logs, to running the detections.

Step 1 — Generate the logs

cd splunk-siem-lab
python3 generate_logs.py
ls logs/          # auth.log, access.log, firewall.log

This uses only the Python standard library — no dependencies or virtual environment are required. It is fully local and does not need Splunk.

Step 2 — Install and start Splunk

Download Splunk Enterprise (a free tier is available from Splunk) and extract the archive:

tar xvzf splunk-*-linux-amd64.tgz     # extracts to ./splunk/
./splunk/bin/splunk start --accept-license

On first start, Splunk prompts for an administrator username and password. When startup completes it prints a web URL, typically http://127.0.0.1:8000.

Step 3 — Open Splunk Web

Browse to http://localhost:8000 and log in with the administrator credentials set in the previous step. (On WSL, the port is forwarded to the Windows host automatically, so a Windows browser reaches it directly.)

Step 4 — Create a dedicated index

Keeping the lab data in its own index isolates it from any other data.

• In the UI: Settings → Indexes → New Index, name it security_lab, accept the defaults, and save.
• Or from the command line: ./splunk/bin/splunk add index security_lab

Step 5 — Ingest the three log files

For each file, use Settings → Add Data → Upload and assign the source type and index:

1. auth.log — source type linux_secure (usually auto-detected); index security_lab.
2. access.log — source type access_combined (Apache combined format, auto-detected); index security_lab.
3. firewall.log — a custom key=value format; assign a source type of firewall (or leave it automatic). Splunk's automatic key=value extraction parses fields such as src_ip, dst_ip, dst_port, action, and bytes; index security_lab. Note that the exact source-type name Splunk stores may differ in capitalization (for example Firewall) depending on how it is assigned; the firewall detections match case-insensitively, but if one returns no results, confirm the stored name with index=security_lab | stats count by sourcetype.

For repeated runs, configuring Splunk to monitor the logs/ directory (Settings → Data inputs → Files & directories) more closely mimics how a production SIEM continuously tails log files, rather than uploading them once.

Step 6 — Confirm the data landed

In the search bar:

index=security_lab | stats count by sourcetype

All three source types should appear with their event counts.

Step 7 — Run the detections

Open detections/SPL_DETECTIONS.md and run each search. Beginning with the brute force detection (below) is recommended, as it demonstrates both aggregation and the correlation principle in a single, clearly verifiable case.

---

6. How the detections work

This section summarizes the detection logic for each scenario. The complete, copy-pasteable SPL for all of them lives in detections/SPL_DETECTIONS.md.

Detection 1 — SSH brute force with success correlation (T1110, T1078)

First, count failed authentications per source IP and flag any source exceeding a threshold well above normal user error:

index=security_lab sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| stats count by src_ip
| where count > 20

Then — the correlation step — check whether a flagged source also produced a successful login, which escalates the finding from attempted to successful compromise. A handful of failures is ordinary mistyping; dozens from one source followed by a success is the signature of a successful password-guessing attack.

Detection 2 — Web directory traversal and scanning (T1190, T1595)

Two angles are combined. A behavioural/content search flags requests for files that legitimate users never request through a web server (for example /etc/passwd, /.env, /.git/config). A signature search flags known scanning tools by their user-agent string. Legitimate traffic does neither, so both are high-signal.

Detection 3a — Port scan (T1046)

Aggregate connections by source and destination and count the number of distinct destination ports each source touched:

index=security_lab sourcetype=firewall action=deny
| stats dc(dst_port) as ports_hit by src_ip, dst_ip
| where ports_hit > 10

A normal client uses a few ports on a server; a source touching many distinct ports on one host — particularly when the connections are denied — is scanning for open services.

Detection 3b — Data-exfiltration beacon (T1041, T1071)

A beacon is the regular, automated check-in that compromised malware sends to an attacker-controlled server — to receive commands or to send stolen data out — on a fixed schedule, by analogy to a lighthouse's steady repeating pulse. Two properties identify this exfiltration. Volume: aggregating outbound bytes per source/destination pair surfaces a host shipping unusually large amounts of data to a single external address. Regularity: plotting the traffic over time with timechart reveals a near-perfect fixed interval (for example every five minutes), which is machine-driven beaconing rather than human activity. The regularity is often the stronger signal — it is not the size alone, but the rhythm.

---

7. Validation results

All four scenarios were validated end to end in a live Splunk Enterprise instance. The generated dataset (293 events: 106 linux_secure, 89 access_combined, 98 firewall) was ingested into the security_lab index, and every detection fired correctly with no false positives:

Detection	ATT&CK	Result
1a — Brute force volume	T1110	203.0.113.66 flagged with 40 failed logins; benign users excluded by the threshold
1b — Success correlation	T1078	Same source: 40 failed, 1 accepted — a confirmed successful compromise
2a — Sensitive-file access	T1190	6 events from 198.51.100.23 targeting /etc/passwd, /etc/shadow, /.env, /.git/config, /backup.sql, win.ini
2b — Scanner user-agent	T1595	198.51.100.23 identified by the sqlmap/1.7 user-agent, 9 requests
3a — Port scan	T1046	198.51.100.23 touched 16 distinct ports on 10.0.1.15
3b-i — Exfil volume	T1041	10.0.1.15 → 203.0.113.200, 12 transfers totalling ~28 MB
3b-ii — Beacon periodicity	T1071	Transfers at an exact 5-minute interval — the machine-driven rhythm characteristic of C2 beaconing

One instructive issue surfaced during validation: the success-correlation search (1b) initially returned nothing when filtering on src_ip=203.0.113.66, because src_ip is a field created by rex only within search 1a and does not persist to other searches. Searching the raw event text ("203.0.113.66") resolved it. This is exactly the kind of field-scoping subtlety that appears only when detections are run against ingested data rather than reasoned about in the abstract, and the corrected form is reflected in detections/SPL_DETECTIONS.md.

---

8. Design notes and limitations

This is a self-contained lab built on synthetic logs with deliberately planted attacks. It demonstrates writing and reasoning about SPL detections and mapping them to MITRE ATT&CK; it does not reproduce the scale or noise of a production SIEM handling real alert volume. The synthetic data is intentionally clean, which makes every detection verifiable but also easier than real-world telemetry, where tuning to suppress false positives is a substantial part of the work.

The lab pairs naturally with two adjacent approaches:

• Metrics-based monitoring (for example a Prometheus/Grafana stack) is optimized for rates and thresholds over time, whereas this log-based SIEM approach is optimized for forensic detail and free-text search across events. The two are complementary halves of security observability.
• Network intrusion detection (Suricata, Snort, Zeek with the Elastic Stack) covers packet- and flow-level detection; this lab adds the Splunk-specific SPL layer on top of the same log-analysis foundation.

The behavioural detections here — flagging unusual volume or regularity — are structurally the same reasoning as statistical anomaly detection in machine learning (for example z-score outlier detection): detection engineering and ML anomaly detection are, in large part, the same problem expressed in different tools.

---

9. Requirements

• Linux environment (including WSL on Windows).
• Python 3.10+ (standard library only) to generate the logs.
• Splunk Enterprise (free tier is sufficient) to ingest and search.

---

10. Possible extensions

• Add a Splunk dashboard (Dashboard Studio or XML) pinning the key detections into a single triage view.
• Add props.conf / transforms.conf definitions for the firewall source type so field extraction is reproducible rather than relying on automatic parsing.
• Extend the log generator with additional sources (for example Windows event logs) and corresponding detections.
• Convert the detections to scheduled alerts to simulate real-time notification.

---

License

Released under the MIT License. See LICENSE for details.