Demon: tmp-dotdir-watcher
Crate: demon-tmp-dotdir-watcher (v0.1.0)
Edition / MSRV: Rust 2021 / 1.74
Host target: single Linux host running systemd
Cadence: every 10 minutes (systemd timer)
Severity: high — would have detected an Azazel-family
compromise on an affected host 71 hours earlier than
operator-flagged 502 detection
A host-resident Rust daemon that scans shallow dot-directories
on a Linux box (/tmp, /home, /var/tmp) every ten minutes,
matches their contents against a SHA-256 IOC list, and
auto-quarantines any hit by stripping permissions (chmod 000)
on the directory. Anything that escapes the allowlist but does
not match an IOC is logged at WARNING for operator triage.
Hidden Azazel-style malware drops everything it needs under
directories whose names start with a dot and live close to the
filesystem root: .r.rpk/, .xdiag/, .perf.c/, .apid/,
or .atmp/. systemd-tmpfiles cleanup does not touch them, and
they sit well inside the noise floor of a busy /tmp/. This
daemon is a bounded, idempotent detector that surfaces them on
the same day they appear.
The canonical operator-facing description (problem statement,
algorithm, configuration, failure modes) lives in ORIGIN.md.
This README is the on-ramp: build, install, run, monitor.
| Event | Severity | Side effect |
|---|---|---|
Dotdir candidate in scan window matches a SHA-256 in /etc/tmp-watcher.iocs |
CRITICAL (PRIORITY=2) |
chmod 000 <path> (idempotent), journal line, log line, NTFY push (when configured in code) |
| Dotdir candidate in scan window is not on the allowlist and not an IOC | WARNING (PRIORITY=4) |
journal line, log line |
| Dotdir candidate matches the allowlist | silent | counters only |
| Anything else | silent | counters only |
The NTFY push side effect is wired through
runtime::push_ntfy_for_match. In the current release that
function calls output::ntfy_push(self.cfg.actions.ntfy_url .as_deref(), …); the URL is read from actions.ntfy_url in
Config (default None, operator-supplied via env or
/etc/tmp-watcher.toml). When unset, the IOC-match alert goes
to the journal and the log file only; when set, the per-tick
summary is POSTed alongside. See
NTFY alert channel below for the env
override + config field + contract doc.
NTFY alert channel — wired (since v0.2.0, DE-018..DE-022
wave). Set DEMON_ACTIONS__NTFY_URL=https://ntfy.sh/<topic>
or actions.ntfy_url: "https://ntfy.sh/<topic>" in
/etc/tmp-watcher.toml. After every tick the daemon POSTs
the per-tick summary with NTFY headers (Title, Priority,
Tags) and text/plain body per
docs/contracts/webhook-payload.md.
Severity → priority mapping: info → 2, warn → 3, error → 5.
A tick_err = true from runtime.run_once short-circuits to
Severity::Error so the operator phone receives a priority-5
alert when the daemon itself fails.
The quarantine is reversible: chmod 700 <path> re-enables the
directory. The daemon never deletes files. See ARCHITECTURE.md
(invariant 7) and RUNBOOK.md §4 for the rollback flow.
The daemon is a single Rust binary. The crate name is
demon-tmp-dotdir-watcher and the installed binary name is the
same.
# toolchain check
cargo --version # 1.74+ required (Cargo.toml `rust-version`)
# release build
cargo build --release
# the binary lands at target/release/demon-tmp-dotdir-watcher
ls -l target/release/demon-tmp-dotdir-watcherBuild-only checks (no host installation needed):
cargo check --all-targets # compile-only, fastest signal
cargo clippy --all-targets -- -D warnings
cargo test # runs unit + integration tests
cargo test --test smoke # CLI smoke test (`--help`, dry-run)
cargo test --test runtime # walk + classify + quarantine e2eThe binary supports three operating modes selected by CLI
flags. The positional [CONFIG_PATH] argument is optional; when
omitted, the embedded default config is used.
# 1. Help
demon-tmp-dotdir-watcher --help
# 2. Validate a config file (no scanning, no side effects)
demon-tmp-dotdir-watcher --validate-config /etc/tmp-watcher.toml
# 3. One polling tick, no quarantine side effect (operators use
# this to sanity-check config + scan coverage + IOC matching
# before activating the daemon)
demon-tmp-dotdir-watcher --dry-run /etc/tmp-watcher.toml
# 4. Production run driven by systemd timer
demon-tmp-dotdir-watcher /etc/tmp-watcher.tomlFlags:
| Flag | Effect |
|---|---|
--help, -h |
Print CLI help and exit 0 |
--validate-config [PATH] |
Load + validate the config; exit 0 on success, non-zero on validation failure |
--dry-run [PATH] |
Load config and run one full polling tick; log JSON events to stderr; never quarantine |
[CONFIG_PATH] (positional) |
Path to YAML config. Falls back to the embedded default |
CLI arguments and the embedded default config cover most deployments. Environment variables override individual fields without editing the config file (see Configuration).
The daemon reads a single TOML file at the path given on the
command line. When no path is given, the embedded default
(config/default.toml, baked into the binary at build time)
is used. Any missing section falls back to the embedded
default; only the keys you actually set override.
Fields the daemon validates on load:
paths.scan_rootsmust be non-empty.paths.scan_maxdepthmust be<= 3.paths.scan_window_minutesmust be> 0.allowlist.max_files_per_dirmust be<= 10.
A reference config matching the embedded default:
# /etc/tmp-watcher.toml
log = { level = "info" }
[runtime]
shutdown_timeout_sec = 30
[paths]
scan_roots = ["/tmp", "/home", "/var/tmp"]
scan_maxdepth = 3
scan_window_minutes = 1440
[ioc]
ioc_list = "/etc/tmp-watcher.iocs"
# ioc_archive_ref = "<path>" # optional; set per-host in /etc/tmp-watcher.toml when forensic auto-refresh is enabled
[allowlist]
allowlist = "/etc/tmp-watcher.allowlist"
max_files_per_dir = 10
[actions]
quarantine_on_ioc_match = true
alert_on_unknown = trueThe journal tag (tmp-watcher), the log file destination
(/var/log/tmp-watcher.log), and the alerts channel
(PRIORITY=2 journal + planned NTFY push) are baked into the
binary and are not configurable from the YAML. Operators tune
behaviour by editing the keys above; tuning the alert
transport requires a code change.
Environment overrides take precedence over the YAML file and
use double-underscore (__) as the section separator:
| Variable | YAML field |
|---|---|
DEMON_LOG_LEVEL |
log.level (string) |
DEMON_SHUTDOWN_TIMEOUT_SEC |
runtime.shutdown_timeout_sec (integer) |
DEMON_PATHS__SCAN_MAXDEPTH |
paths.scan_maxdepth (integer) |
DEMON_PATHS__SCAN_WINDOW_MINUTES |
paths.scan_window_minutes (integer) |
DEMON_PATHS__SCAN_ROOTS |
paths.scan_roots (colon-separated list) |
DEMON_IOC__IOC_LIST |
ioc.ioc_list (path) |
DEMON_IOC__IOC_ARCHIVE_REF |
ioc.ioc_archive_ref (path) |
DEMON_ALLOWLIST__ALLOWLIST |
allowlist.allowlist (path) |
DEMON_ALLOWLIST__MAX_FILES_PER_DIR |
allowlist.max_files_per_dir (integer) |
DEMON_ACTIONS__QUARANTINE_ON_IOC_MATCH |
actions.quarantine_on_ioc_match (bool) |
DEMON_ACTIONS__ALERT_ON_UNKNOWN |
actions.alert_on_unknown (bool) |
DEMON_ACTIONS__NTFY_URL |
actions.ntfy_url (URL string; per-tick summary emit) |
An env override that fails to parse is logged at WARN and ignored; the TOML value (or the embedded default) is kept.
Validate after any edit:
demon-tmp-dotdir-watcher --validate-config /etc/tmp-watcher.toml/etc/tmp-watcher.iocs — one SHA-256 per line. The matcher
hashes the first max_files_per_dir files inside each
candidate dotdir and looks up the hash string. Lines that are
blank or start with # are ignored.
# Example IOC entry (synthetic hash for documentation; not a real sample)
b02ad43cfa407a01c376c7a904104b03 trunk.md5
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 trunk.sha256
A missing IOC list is treated as an empty list — the daemon boots, logs a warning, and continues scanning. This is the documented behaviour for dev boxes; production deploys must keep the file populated and review it against the canonical forensic archive (operator-supplied path via per-host config).
/etc/tmp-watcher.allowlist — one glob per line (the globset
syntax). Anything matching a glob is skipped silently. Comment
lines (# ...) and blank lines are allowed.
# FastPanel2 X11 / systemd-private entries — legitimate, keep
.font-unix
.ICE-unix
.X11-unix
.XIM-unix
.Test-unix
systemd-private-*
A missing or unreadable allowlist becomes an empty in-memory allowlist; a WARNING is logged and scanning proceeds. Operators expand this file as new false-positive patterns are observed; the daemon picks up changes on the next poll cycle (within ten minutes), no restart needed.
| Metric | Value | Notes |
|---|---|---|
| Binary (main) | 6.7 MB | demon-tmp-dotdir-watcher, stripped release |
| Binary (sidecar) | 2.3 MB | demon-tmp-watcher-cross-host, one-shot |
| RSS idle | 7.0 MB | boot complete, between ticks |
| RSS peak (500 candidates) | 7.3 MB | +264 KB ≈ 540 B per candidate |
| Tick duration (500 dotdirs) | ~5 ms | measured on a typical tmpfs/ext4 |
| Virtual size | ~764 MB | unmapped mmap; not a RAM concern |
The daemon holds no state between ticks: the IOC HashSet, allowlist
GlobMatcher set, and per-tick Vec<Candidate> are released at the
end of every poll cycle. Steady-state RSS stays at 7.0 MB across
thousands of cycles (no leak). Memory growth is linear in candidate
count and bounded by scan_maxdepth × max_files_per_dir per
scan_root.
Run the daemon under Type=oneshot systemd activation so the process
is reclaimed between activations — the daemon does not need to hold
memory when no timer is firing.
The daemon is timer-driven. The stub unit + timer ship in
contrib/systemd/; install via make install-units
(copies to /etc/systemd/system/) or copy them manually.
# One-shot install: build + ship binary, units, and a fresh
# /etc/tmp-watcher.toml if one is not already present. The
# /etc/tmp-watcher.toml copy is idempotent (preserves operator
# edits on re-install). Per the daemon's "proposed" status, the
# systemd units are installed but NOT enabled — first-deployment
# enablement is operator-side.
make installThe full manual layout (drop-in):
/etc/systemd/system/tmp-watcher.service
[Unit]
Description=demon-tmp-dotdir-watcher (shallow dot-directory IOC scanner)
After=local-fs.target
Documentation=file:<repo>/README.md
[Service]
Type=oneshot
ExecStart=/usr/local/bin/demon-tmp-dotdir-watcher /etc/tmp-watcher.toml
WorkingDirectory=/var/lib/tmp-watcher
StateDirectory=tmp-watcher
LogsDirectory=tmp-watcher
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=false # the daemon must read /tmp; private /tmp would hide signals
ReadWritePaths=/run /var/log /var/tmp /tmp /home
CapabilityBoundingSet=
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=tmp-watcher
[Install]
WantedBy=multi-user.target/etc/systemd/system/tmp-watcher.timer
[Unit]
Description=demon-tmp-dotdir-watcher poll timer
[Timer]
OnBootSec=2min
OnUnitActiveSec=10min
AccuracySec=30s
Persistent=true
Unit=tmp-watcher.service
[Install]
WantedBy=timers.targetOne-time install on the target host (assumes you have
tmp-watcher.toml, tmp-watcher.allowlist, and
tmp-watcher.iocs already authored somewhere on the build
host — copy them in via your normal config-management path):
sudo install -m 0755 target/release/demon-tmp-dotdir-watcher \
/usr/local/bin/demon-tmp-dotdir-watcher
sudo install -m 0644 tmp-watcher.toml /etc/tmp-watcher.toml
sudo install -m 0644 tmp-watcher.allowlist /etc/tmp-watcher.allowlist
sudo install -m 0644 tmp-watcher.iocs /etc/tmp-watcher.iocs
# sanity-check that the config the daemon will see parses + validates
demon-tmp-dotdir-watcher --validate-config /etc/tmp-watcher.toml
# activate the poll timer
sudo systemctl daemon-reload
sudo systemctl enable --now tmp-watcher.timerRestart=on-failure (never always) is intentional: the
daemon is timer-driven and a hard restart loop on a persistent
failure would burn CPU.
Visibility caveat:
chmod 700 /home/<user>interacts withCapabilityBoundingSet=(CR-006). The recommended hardening (ProtectHome=read-only,CapabilityBoundingSet=empty) drops all capabilities and grants the daemon only "others" access on filesystem entities it does not own or have a group grant for. On a host withchmod 700 /home/<user>directories, the daemon cannot traverse those subtrees — the/homearm of the threat model (Azazel's secondary footprint under.docker,.ssh,.cache,.rpk,.xdiag, etc.) becomes invisible to the walker. The daemon does NOT silently swallow this: each unreadable scan root produces a syntheticCandidate { skipped_reason: Some(IoError) }, incrementsRunSummary.skipped, and surfaces the path viaRunSummary.unreadable_rootson the per-poll summary log line (journalctl -t tmp-watcher ... | grep unreadable_roots).Operator remediation options:
- Remove
/homefromscan_rootsand rely on/tmp+ overlay-scan arms. Accept the visibility loss.- Add the daemon user's gid to those homes' owning group so
read_dirsucceeds on the directory execute bit.- Switch
ProtectHome=tofalse(drops the kernel protection that caused the EACCES) and re-evaluate the threat model trade-off.The daemon's default config keeps
/homeinscan_roots; the choice is operator-facing and documented above.
| Where | What | Filter |
|---|---|---|
journalctl -t tmp-watcher -n 50 |
last 50 structured events | service journal |
journalctl -t tmp-watcher PRIORITY=2 -S -24h |
CRITICAL (IOC match) | last 24h |
journalctl -t tmp-watcher PRIORITY=4 -S -24h |
WARNING (unknown dotdir) | last 24h |
tail -F /var/log/tmp-watcher.log |
full event log with timestamps | host log |
| NTFY push (per-tick summary) | wired through output::push_tick_summary when Config.actions.ntfy_url is set; contract at docs/contracts/webhook-payload.md |
operator phone (when configured) |
The daemon uses tracing-subscriber with JSON output; each
event carries candidates, allowlisted, ioc_matches,
quarantined, and skipped counters when emitted by
--dry-run or at end of poll.
The minimum-viable health check for a deployment:
# timer scheduled?
systemctl list-timers --all | grep tmp-watcher
# last run status?
systemctl status tmp-watcher.service
# recent activity?
journalctl -t tmp-watcher -n 50 --since "1 hour ago"
# CRITICAL events in the last 24h?
journalctl -t tmp-watcher PRIORITY=2 -S -24h
# WARNING events in the last 24h?
journalctl -t tmp-watcher PRIORITY=4 -S -24hFull operator triage — what to do on CRITICAL, how to roll
back a false-positive quarantine, what to capture before
escalating — lives in RUNBOOK.md. It is built around the
exact paths, journal tags, and quarantine side effect described
above, so the two documents stay in sync.
demon-tmp-dotdir-watcher/
├── Cargo.toml # crate manifest, deps, MSRV 1.74
├── config/
│ └── default.toml # embedded default config (bake-in via include_str!)
├── src/
│ ├── main.rs # CLI parser, boot, signal handling
│ ├── config.rs # YAML loader + validation + env overlay
│ ├── runtime.rs # main poll loop, RunSummary, quarantine wiring
│ ├── subsystem.rs # walk() + walk_decision_pipeline() + quarantine()
│ ├── allowlist.rs # glob-based allowlist filter
│ ├── ioc.rs # IOC loader + SHA-256 matcher + hash_file()
│ ├── output.rs # structured event emitters (PRIORITY=2, =4)
│ └── test_util.rs # shared TempDir/TempFile for unit tests
├── tests/
│ ├── smoke.rs # `--help` + `--validate-config` smoke
│ └── runtime.rs # walk + classify + quarantine e2e
├── contrib/
│ ├── config/
│ │ └── tmp-watcher.conf.example # operator-facing example config (TOML format, peer-daemon .conf extension)
│ └── systemd/
│ ├── tmp-watcher.service # Type=oneshot stub per invariant 2
│ └── tmp-watcher.timer # OnUnitActiveSec=10min
├── docs/
│ ├── components/tmp-watcher.md
│ ├── contracts/
│ │ ├── tmp-watcher-allowlist-ioc.md
│ │ └── webhook-payload.md # NTFY post-tick summary contract
│ ├── architecture/ROADMAP.md
│ └── architecture/STATUS.md
├── ORIGIN.md # canonical operator-facing description
├── DAEMON.md # on-ramp summary for an architect
├── ARCHITECTURE.md # component breakdown + invariants + failure modes
├── Makefile # build / test / lint / install helpers
├── RUNBOOK.md # operator triage flow + Webhook channel triage
└── README.md # this file
contrib/ships the operator-side artifacts (example config + stub systemd units) perDemon/docker-janitorpeer convention; theMakefiletargetsinstall-config/install-bin/install-unitsinstall them in one shot.make enableis intentionally NOT defined — the daemon is operator-side activated per its "proposed" status in Current state.
## Where to look next
- `ORIGIN.md` — full problem statement and algorithm. The
authoritative source for what the daemon does and when.
- `DAEMON.md` — on-ramp summary covering the six things every
architect should know before touching this daemon
(forensic origin, host-side state paths, journal tags,
restart policy, quarantine side effect).
- `ARCHITECTURE.md` — components, invariants (`Idempotent
restart`, `No silent background timers`, `Quarantine is
reversible`, etc.), failure modes, Rust-port migration
checklist.
- `RUNBOOK.md` — operator triage for CRITICAL, WARNING,
not-running, and quarantine rollback; restart and reload
policy.
- `docs/architecture/STATUS.md` — current build / wave state.
- `docs/components/tmp-watcher.md` — component-level view
(purpose, inputs/outputs, invariants, failure modes).
- `docs/contracts/tmp-watcher-allowlist-ioc.md` — file shape
contract for the allowlist and IOC list (required by the
loader; do not change without updating this contract).
- ORIGIN.md § "Problem it solves" — the originating incident that
informed the original spec (Azazel-family malware in shallow
dot-directories). Reference for the threat model.
## Source of truth
`ORIGIN.md` is the canonical operator-facing description. Any
disagreement between this folder and `ORIGIN.md` is a bug;
reconcile by updating this folder to match.