Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,19 @@ updates:
prefix: chore
include: scope

# The base image of the SSH targets of the end-to-end test, pinned by
# its digest. Dependabot updates the digest. Pull request titles look
# like "test(deps): bump debian ...".
- package-ecosystem: docker
directory: /test/e2e/testdata/sshd
schedule:
interval: weekly
cooldown:
default-days: 7
commit-message:
prefix: test
include: scope

# Pull request titles look like "ci(deps): bump actions/checkout ...".
# Dependabot also updates the pinned commit SHA and its version comment.
- package-ecosystem: github-actions
Expand Down
30 changes: 30 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,36 @@ jobs:
with:
# Dependabot does not update this version. Update it by hand.
version: v2.14.0
# With the tag e2e, the files of the end-to-end test are checked
# too. No file needs the tag to be missing, so nothing is left out.
args: --build-tags e2e

e2e:
name: end-to-end test
runs-on: ubuntu-latest
# The test itself takes about 10 seconds; without caches, building
# the image and the binaries takes a few minutes more.
timeout-minutes: 20
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: go.mod
- name: Show the programs that the test uses
# The runner has Docker, the OpenSSH client and git. The test
# needs OpenSSH 8.9 or newer.
run: |
docker version
ssh -V
git --version
- run: go vet -tags e2e ./test/e2e/...
- name: End-to-end test
# See DESIGN.md section 17. The test builds nodkey and the image
# of test/e2e/testdata/sshd, and removes every container, network
# and image that it makes. It fails if one is left.
run: go test -race -tags e2e ./test/e2e/...

vuln:
runs-on: ubuntu-latest
Expand Down
16 changes: 9 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,23 +28,25 @@ Planned. Each v1 issue adds its part.
(sections 6 and 15) and `nodkey ssh-grant` (sections 13 and 15).
- `contrib/systemd/` - the example systemd unit of the broker, with
encrypted credentials and a sandbox (DESIGN.md section 16).
- `test/e2e/` - end-to-end tests with `sshd` in a container. Build tag
`e2e`.
- `test/e2e/` - the end-to-end test with `sshd` in a container (DESIGN.md
section 17). Build tag `e2e`. The image of the container is
`test/e2e/testdata/sshd`.
- `.claude/agents/security-reviewer.md` - the security review every change
gets.

## Commands

These work once the Go module exists (issue #2); the e2e test comes with
issue #15.

```sh
go test -race ./...
go vet ./...
golangci-lint run
go test -race -tags e2e ./test/e2e/... # needs Docker
golangci-lint run --build-tags e2e # the tag adds the files of test/e2e
go test -race -tags e2e ./test/e2e/... # needs Docker, ssh and git
```

The last one is the end-to-end test (DESIGN.md section 17). It builds
nodkey and a Docker image, and removes every container, network and image
that it makes.

## Rules

- **This repository is public.** Never add a real host name, domain name,
Expand Down
103 changes: 91 additions & 12 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -2490,18 +2490,97 @@ bws 2.1.0 and bw 2026.9.1 on 2026-10-14
`bw` when it runs under that name. It records its arguments and its
environment, so the tests can check that no token, password or session
key is ever an argument.
- **End-to-end test**: a throwaway key, `sshd` in a container, the broker and
the client on the test machine, and the fake approver. It checks that
`ssh` and `git` over SSH work, that a deny blocks the login, that an
unknown host key is refused, that `ssh-grant` covers exactly the listed
hosts, and that `nodkey run` passes a secret into a command without
printing it. It needs Docker, so it has its own build tag and its own CI
job.
- **Canary test**: the end-to-end test uses a planted secret value and then
searches every log, the audit log, every message sent to the fake
Telegram server, and the client's output for it. A second canary sits in
a parent process's command line, to test the masking. Any hit fails the
test.
- **End-to-end test** (`test/e2e`): the real programs, with only Telegram
and the store faked. It builds the `nodkey` binary from the repository,
with the race detector when the test runs with it.
- The SSH targets run in one Docker container, built from
`test/e2e/testdata/sshd`: OpenSSH's `sshd` and git from Debian, on a
base image pinned by its digest. Each target is an `sshd` of its own,
with its own throwaway host key: `web-1` (Ed25519), `web-2` (ECDSA),
`db-1` (RSA) and `web-3` (Ed25519). The broker's `known_hosts` has the
first three, not `web-3`. The remote users `deploy` and `ops` have the
public half of the broker's throwaway key `deploy` in their
`authorized_keys`. Only the broker holds its private half. `deploy`
has a bare git repository. The container has no capability but those
that `sshd` needs, no new privileges, limits on its processes and
memory, and `DisableForwarding yes`. It publishes its ports on
`127.0.0.1` only, from a network of its own.
- `nodkey broker`, `nodkey run`, `nodkey ssh-agent`, `nodkey ssh-grant`,
`ssh` and `git` run on the test machine. The files of the broker and
the clients are in a private directory that passes every check of
section 14, with TLS keys from `nodkey keygen`. The store is a fake
`bws`, a shell script. The broker's Telegram code is the real one: it
reaches a fake Telegram server through a fake HTTPS proxy and a test
CA, as in the tests of `nodkey broker`. The test taps "Approve once",
"Approve 15 min" or "Deny" there. Two clients, `laptop` and `ci`, have
their own keys and home directories.
- The cases: `nodkey run` passes the secret into a command, which proves
it with the SHA-256 of the value; after a Deny, the command does not
run; an SSH login after "Approve once"; an unknown host key is refused
before any message; after `ssh-grant` for `web-1` and `web-2`, these
hosts need no tap, and `db-1` and the remote user `ops` still ask;
`git clone` and `git push` over SSH, where one "Approve 15 min" covers
the logins after it; a denied login, after which the grants of the key
have ended. Each case also checks what the message shows, what the
clients print, the entries in the audit log, and that no other
message goes out. The config has no "never ask" rule, and nothing in
the test makes an alert, so every message must have buttons. At the
end, `nodkey audit verify` checks the whole audit log with every
anchor of the broker's journal, and a copy of the log with one changed
entry fails.
- The root-only test of `internal/client` runs as root in a container
without a network, with only the capabilities `SETUID` and `SETGID`:
a process of another user connects to the socket of
`nodkey ssh-agent`, and the agent refuses it (`SO_PEERCRED`).
- Run it with `go test -race -tags e2e ./test/e2e/...`. It needs
Docker, OpenSSH's `ssh` 8.9 or newer, and `git`. Once the image and
the binaries are built, it takes about 10 seconds. The CI job `e2e`
runs it.
- Each run of the test has a run ID: its start time and a random part.
The names and labels of its containers, its network and its image
hold it, and so does the name of its private directory, in
`$XDG_RUNTIME_DIR`, or else in the cache directory (`$XDG_CACHE_HOME`,
or else `~/.cache`). While it runs, the test holds a lock on the file
`nodkey-e2e-RUNID.lock` in the temporary directory (`$TMPDIR`, or else
`/tmp`). At the end, it removes everything that it made, and fails if
something is left. A run that is killed, such as by the `-timeout` of
`go test` or by Ctrl-C, cannot do that: the kernel stops its programs,
its containers end on their own after 15 minutes, and the rest stays.
So each run first removes
what runs that have ended left: those whose lock file nobody holds,
and those older than an hour whose lock file it cannot check: it is
not there, or not a regular file. It never touches a run that holds
its lock, or anything without a run ID. By hand, when no end-to-end
test runs:

```sh
docker ps --all --quiet --filter label=nodkey-e2e | xargs -r docker rm --force
docker network ls --quiet --filter label=nodkey-e2e | xargs -r docker network rm
docker image ls nodkey-e2e-sshd --format '{{.Repository}}:{{.Tag}}' | xargs -r docker image rm
rm -rf "${XDG_CACHE_HOME:-$HOME/.cache}"/nodkey-e2e-* "${TMPDIR:-/tmp}"/nodkey-e2e-*.lock
if [ -n "$XDG_RUNTIME_DIR" ]; then rm -rf "$XDG_RUNTIME_DIR"/nodkey-e2e-*; fi
```
- **Canary test**: part of the end-to-end test. The first canary is the
value of the secret in the store. The second canary is a random word in
the command line of a wrapper process, which runs `ssh` and `git`: as
the value of `--api-token=`, and as a word of its own. The test searches
every log (the broker's journal, `nodkey ssh-agent` and the `sshd`s),
the audit log, every call to the fake Telegram server, and every output
of the clients. It looks for the first canary as it is, in lower and
upper case, in hex, and in base64 and base64url at each offset, and for
each part of 12 characters of its random part in the same forms, so
that a value that was cut is found too. It looks for the made-up bot
token and the token of the store in the same forms, and for the second
canary in any case. Any hit fails the test. The command line of the
wrapper is claimed text, so section 7 shows it only in the message and
in the audit log, masked. The test checks that every message and every
audit entry of an SSH login shows `***` in place of both forms. It also
checks that no other call to Telegram, and no output of `nodkey`, holds
claimed text: a reason, a word of a command line, the working
directory, or the variable name. Guards make sure that there was
something to find: the broker handed out the secret (a `secret_sent`
entry), and SSH logins went through the wrapper. So a `-run` filter
that leaves out the cases makes the search fail, not pass.
- **One manual test with a real Telegram bot**, written down in
`docs/manual-telegram-test.md`. The token comes from a local file and is
never committed.
Expand Down
140 changes: 140 additions & 0 deletions test/e2e/audit_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
//go:build e2e

package e2e

import (
"bytes"
"encoding/json"
"errors"
"io/fs"
"os"
"slices"
"strconv"
"strings"
"testing"
"time"
)

// auditLog reads the audit log of the broker, one case after the other:
// each case checks the entries that it caused.
type auditLog struct {
path string
seen int // the entries that the cases have checked
}

// entry is one entry of the audit log.
type entry map[string]any

// field returns the field k as text: a string as it is, a list of strings
// joined with ",", a number in decimal, and "" when the entry does not
// have it.
func (en entry) field(k string) string {
switch v := en[k].(type) {
case string:
return v
case float64:
return strconv.FormatFloat(v, 'f', -1, 64)
case []any:
parts := make([]string, len(v))
for i, p := range v {
parts[i], _ = p.(string)
}
return strings.Join(parts, ",")
case nil:
return ""
}
return "(not a string, a number or a list)"
}

// name returns the event of the entry, with its answer, cause, via or
// code, such as "decision:approve_once" or "denied:not_allowed".
func (en entry) name() string {
name := en.field("event")
for _, k := range []string{"answer", "cause", "via", "code"} {
if v := en.field(k); v != "" {
name += ":" + v
}
}
return name
}

// eventNames returns the names of entries.
func eventNames(entries []entry) []string {
out := make([]string, len(entries))
for i, en := range entries {
out[i] = en.name()
}
return out
}

// entries returns the complete lines of the audit log, as entries. The
// broker writes each line with one write call, but a read can come in the
// middle of it, so a last line without its newline is left for later.
func (a *auditLog) entries(t *testing.T) []entry {
t.Helper()
data, err := os.ReadFile(a.path)
switch {
case errors.Is(err, fs.ErrNotExist):
return nil // the broker makes it at its start
case err != nil:
t.Fatal(err)
}
data = data[:bytes.LastIndexByte(data, '\n')+1]
var out []entry
for line := range strings.Lines(string(data)) {
var en entry
if err := json.Unmarshal([]byte(line), &en); err != nil {
t.Fatalf("audit log line %d is not JSON: %v", len(out)+1, err)
}
out = append(out, en)
}
return out
}

// next waits until the audit log has len(want) entries after those that
// the cases have checked, at most settleLimit, and returns them. The
// broker writes the entries of a request before it answers the client. It fails unless their names
// are want, in this order, and unless each of them has the fields of
// subject, where "" means that the entry does not have the field. The
// entries of one request also share its request ID.
func (a *auditLog) next(t *testing.T, subject map[string]string, want ...string) []entry {
t.Helper()
deadline := time.Now().Add(settleLimit)
var got []entry
for {
all := a.entries(t)
got = all[min(a.seen, len(all)):]
if len(got) >= len(want) || time.Now().After(deadline) {
break
}
time.Sleep(20 * time.Millisecond)
}
got = got[:min(len(got), len(want))]
a.seen += len(got)
if !slices.Equal(eventNames(got), want) {
t.Fatalf("audit entries %q, want %q", eventNames(got), want)
}
for i, en := range got {
for k, v := range subject {
if g := en.field(k); g != v {
t.Errorf("audit entry %d (%s) has %s %q, want %q", a.seen-len(got)+i+1, en.name(), k, redact(g), v)
}
}
if id := got[0].field("request_id"); en.field("request_id") != id {
t.Errorf("audit entry %d (%s) has another request ID than the entries before it", a.seen-len(got)+i+1, en.name())
}
}
return got
}

// unchecked reports the entries that no case has checked yet, and marks
// them as checked, so that one failed case does not make every later case
// fail.
func (a *auditLog) unchecked(t *testing.T) {
t.Helper()
all := a.entries(t)
if len(all) > a.seen {
t.Errorf("audit entries that no case expected: %q", eventNames(all[a.seen:]))
a.seen = len(all)
}
}
Loading
Loading