Device Test Framework

A working example of on-device integration testing: running an automated test suite directly on embedded/edge hardware using GitHub Actions self-hosted runners and a privileged Docker test container.

If your product is a Linux box running a stack of Docker containers, host services, databases, and web UIs, and you've ever asked "how do I run real integration tests against the actual hardware in CI?" — this repo is a template answer.

                       GitHub Actions (workflow_dispatch / PR)
                                      │
                                      ▼
        ┌──────────────────────── Device (edge hardware) ────────────────────────┐
        │                                                                        │
        │   self-hosted runner ──► test container (node:20-slim, --privileged)   │
        │                             │                                          │
        │        ┌────────────────────┼──────────────────────┐                   │
        │        ▼                    ▼                      ▼                   │
        │   docker.sock          host mounts             nsenter → host          │
        │   (inspect/restart     (/home/device,          (systemctl, hostname,   │
        │    app containers)      /opt/device)            host processes)        │
        │                                                                        │
        │   app containers:  your app services…                                   │
        └────────────────────────────────────────────────────────────────────────┘
                                      │
                                      ▼
                     JUnit results → GitHub checks + artifacts

How it works

1. The device is the CI runner. A GitHub Actions self-hosted runner is installed on the device itself (setup guide). Workflows target it by label.
2. Tests run in a privileged container on the device. The workflow starts a node:20-slim container with --privileged --pid=host, the Docker socket, and the device's app directories mounted. From inside it, tests can:
   • drive the host's Docker daemon (inspect, restart, exec into app containers)
   • read and write files on the host filesystem
   • execute commands in the host's namespaces via nsenter (ShellHelper.execOnHost)
3. Vitest runs the suite sequentially (single worker, generous timeouts, retries) — the right shape for tests that mutate real device state.
4. Results come back as JUnit XML → GitHub check annotations + downloadable artifacts.

⚠️ Security note: the test container intentionally has root-equivalent access to the device. Only install self-hosted runners on lab/test devices you control, keep the repo private if runners are attached, and never expose these runners to workflows from untrusted forks.

What's inside

Helpers (src/helpers/)

Helper	What it does
ShellHelper	Run commands in the test container or on the host (via nsenter), read/parse files, wait for conditions
DockerHelper	List/inspect/restart containers, tail logs, wait for healthy state or log text
MongoHelper	Query a MongoDB container via docker exec + mongosh — no driver or DB port needed
LogParser	Parse JSON logs (pino/bunyan style) and plain-text logs, filter by level/pattern/nested field
test-conditions	Skip tests cleanly when an optional container isn't on the device (describe.skipIf(requireContainer('mongo')))

Log matchers (src/helpers/log-matchers/)

Small, pure functions per service that turn raw log output into domain assertions — write one module per container in your stack (see the pattern):

const logs = DockerHelper.getLogs(containerId, 1000);
expect(MyService.hasStarted(logs)).toBe(true);
expect(MyService.hasErrors(logs)).toBe(false);
expect(MyService.containsSequence(logs, ['database connected', 'my-service started'])).toBe(true);

Test suites (src/__tests__/)

Suite	Demonstrates
smoke.test.ts	Environment sanity: we're on the device, the host is reachable, Docker works
docker/	Container presence & health checks across the whole stack
system/	Asserting on the host OS (systemd units, disk space) from inside the container
web-ui.test.ts	Playwright tests against any web UI reachable from the device (page objects in src/support/)

Getting started

1. Install a self-hosted runner on the device — see docs/self-hosted-runner.md
2. Adjust src/constants/device-paths.ts and the workflow volume mounts to match your device's filesystem
3. Point the helpers and example tests at your own container names and file paths
4. Push, then run the Device Integration Tests workflow

To iterate locally on the device itself, clone the repo there and run npm install && npm run test:smoke directly.

Key design decisions

• Tests run ON the device, not against it over the network. No SSH plumbing inside tests, no flaky network assertions — helpers call docker/mongosh/nsenter directly.
• Sequential execution (one worker, no file parallelism) because tests restart real containers and mutate real state.
• Conditional execution over branching test code — container-presence guards keep one suite valid across devices with different container sets.
• Log-based assertions for services without rich APIs: parse structured logs into domain predicates rather than grepping in every test.
• Everything injected via env (.env locally, secrets/vars in Actions) — no credentials in the repo.

Repository layout

├── .github/workflows/      # On-device CI pipeline
├── docs/                   # Self-hosted runner setup guide
└── src/
    ├── __tests__/          # Test suites by subsystem
    ├── constants/          # Device filesystem paths
    ├── helpers/            # Shell/Docker/DB helpers + log matchers
    ├── support/            # Playwright page objects & browser setup
    └── types/              # Shared types

Adapting this to your product

Everything product-specific in this repo is a placeholder: the paths under /home/device and the example container checks. The pattern — device as runner, privileged test container, helper layer, log matchers — is the part meant to be reused.

License

MIT