diff --git a/DESIGN.md b/DESIGN.md index 1ef9968..5132fdf 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -27,6 +27,9 @@ protects, and where it stops. button. It holds the code and the button (apply or skip). Nothing else. - **tree**: a clean, read-only copy of the files of one commit, written by pullgate itself from git objects. +- **check request**: a small file that the listener writes when the owner + sends `/check` or `/check NAME`. It holds no data from the message. It + starts a normal run now, instead of at the next timer (sections 8 and 9). - **pause file**: `/var/lib/pullgate/NAME/paused`. While it exists, the instance does nothing: no run, no apply. @@ -49,7 +52,9 @@ These rules come from issue #1. Each one has tests (section 14). no port. It accepts a tap only from the configured owner user ID, in the configured chat, on an open question. All it can do is write an approval for that open question. The approval carries only the code. Root looks - up the SHA in its own state. + up the SHA in its own state. The only other thing it can write is a + check request for `/check` of the owner. That request holds no data, and + it only starts a normal run sooner (sections 8 and 9). 5. **Root does the apply.** It acts only on an approval whose code matches a question that root recorded itself, and only within the time limit of that question. By default it moves forward only: the new SHA must descend from @@ -66,7 +71,7 @@ These rules come from issue #1. Each one has tests (section 14). | process | user | started by | holds | |---|---|---|---| -| `pullgate run NAME` | root | a timer (`pullgate-run@NAME.timer`) | the clone, the state, the deploy key | +| `pullgate run NAME` | root | a timer (`pullgate-run@NAME.timer`), or a path unit (`pullgate-check@NAME.path`) when a check request exists | the clone, the state, the deploy key | | `pullgate apply NAME` | root | a path unit (`pullgate-apply@NAME.path`) when the inbox is not empty, or the auto approval exists | the state | | `pullgate listen` | `pullgate` (own user) | a service (`pullgate-listen.service`) | the bot token, its list of open questions | | `check`, `summary` | `check_user` (default `pullgate-check`) | `pullgate run` | nothing | @@ -74,7 +79,9 @@ These rules come from issue #1. Each one has tests (section 14). | `pullgate status NAME` | root | the owner | nothing | Root and the listener never talk directly. They use a spool directory -(section 8). +(section 8). Both units start the same `pullgate-run@NAME.service`, so a +run on `/check` has the same sandbox, the same lock and the same rules as a +run on the timer. systemd never starts one unit twice at once. The run unit is sandboxed (read-only system, private `/tmp`, no new privileges). The apply unit runs the owner's `apply` command as root, so it @@ -99,6 +106,7 @@ can be less strict; each example sets what its `apply` needs. | `/var/spool/pullgate/` and `/var/spool/pullgate/NAME/` | root, 0755 | the spool | | `/var/spool/pullgate/NAME/outbox` | root, group listener, 0770 | messages from root to the listener | | `/var/spool/pullgate/NAME/inbox` | listener, 0700 | approvals from the listener to root | +| `/var/spool/pullgate/NAME/check` | listener, 0700 | check requests from the listener to root | | `/var/lib/pullgate-listener/state.json` | listener, 0600 | the listener's open questions | The paths can be changed with `--config-dir`, `--state-dir` and @@ -121,8 +129,8 @@ path that the listener can change. It opens the spool directories with through the open directory, creates files in them with `O_CREAT|O_EXCL|O_NOFOLLOW` under a random temporary name and renames them in the same directory. `/var/spool/pullgate` and `NAME/` are owned by root -and pass the check of section 6, so the listener cannot swap `outbox` or -`inbox` for a symlink. +and pass the check of section 6, so the listener cannot swap `outbox`, +`inbox` or `check` for a symlink. ## 5. The instance config @@ -266,8 +274,11 @@ used again. ## 7. One run (`pullgate run NAME`) 1. Read the config. Take the lock (`flock`, no wait). If another run or - apply holds it, log "busy" and exit 0. If the pause file exists, log - "paused" and exit 0. If `mode = "off"`, log "off" and exit 0. + apply holds it, log "busy" and exit 0; but when a check request waits, + wait for the lock instead (section 8). If the pause file exists, log + "paused" and exit 0. If `mode = "off"`, log "off" and exit 0. When a + check request waits, take it first (section 8); then paused and off + send a reply ("NAME is paused" or "NAME is off"), and do not fetch. 2. If the state shows an apply in progress (section 10), write the pause file, report, and exit. 3. Make sure the clone and the spool directories exist with the right owners @@ -338,6 +349,59 @@ files older than one minute. auto approval (`PathExists=`), and starts `pullgate apply NAME`. The apply service has a start limit, so a stuck file cannot start it in a loop. +**Check requests** (the listener writes, root reads and removes). The +listener writes `/var/spool/pullgate/NAME/check/request` for `/check` +(section 9): a `.tmp-*` file renamed to that one fixed name, so many +requests before root reads them are one file. The content is always the +six bytes `check\n`. It holds nothing from the message: not the text, not +a SHA, not a code. A check request can only start a normal run. It is not +an approval, and `pullgate apply` never reads this directory. + +`pullgate-check@NAME.path` watches the file `check/request` +(`PathExists=`; another file there starts nothing) and starts `pullgate-run@NAME.service`, the unit of the timer. `pullgate run`, +with the lock held (a run that had to wait for the lock, and then finds +no request, stops: another run took the request): + +1. Reads and removes every file in the directory, with the checks of an + approval: through `os.OpenRoot` with `O_NOFOLLOW`, a regular file owned + by the listener user, at most 64 bytes. It must also have the name + `request` and exactly the content `check\n`. Root removes each file + before it acts. A bad file is logged and removed; it is not a request. + A run that finds only bad files goes on as a normal (timer) run, with + no reply and no mark against the limit. +2. **The limit**, kept by root in its own `state.json` (`check_at`), not + by the listener: at most one run per minute (`CheckInterval`) starts + from a check request. When the last one is less than a minute away + (before or after, so a clock jump back of less than a minute does not + pass it), the run removes the request and exits without a fetch. It + sends one reply per minute: "A check ran less than a minute ago, so this + /check was dropped." (`check_note`). +3. Paused or off: reply so, do not fetch. +4. Else the normal run (section 7). After it, a reply says what it found, + unless the question or a notice says it already: "No new commit. The + applied commit is SHA.", "the question for SHA is still open", "SHA was + handled before (mark)", "the last apply of SHA failed and it is held", + or, in `auto` mode, "New commit SHA … it is applied now, without a tap" + (section 13). A run that fails sends "The check failed. See the journal + of the host", without the error text. + +**A request while the lock is held.** A run started by the timer exits +"busy" when another process holds the lock. A run that sees a check request +waits for the lock instead (`flock` with a wait), so the request is not +lost and the path unit does not start the run again and again. The lock +makes sure that two runs, or a run and an apply, never run at once. The +run unit's `TimeoutStartSec` caps the wait. + +**The start limit.** Each check request starts the run unit. The path unit +has a trigger limit (`TriggerLimitBurst=10` in 60 seconds), which stops a +loop when a run fails before it removes the request. If the limit is hit, +systemd stops the path unit until the owner runs `systemctl reset-failed +pullgate-check@NAME.path` and starts it again. The run unit has no start +limit (`StartLimitIntervalSec=0`), so many requests never stop the timer. +So that `/check` alone never hits the limit, the listener also writes at +most one request per instance in 10 seconds (`CheckGap`, kept in +memory). Root does not trust this: its own limit is the one above. + ## 9. Questions and approvals Root keeps its questions in `state.json`: @@ -355,7 +419,11 @@ still counts (rule 2). The listener forgets a code when it handles the Every list entry is removed when it is answered or expires. **The listener** calls `deleteWebhook` at start and then `getUpdates` with -`allowed_updates = ["callback_query"]`. It accepts a tap only when all of +`allowed_updates = ["callback_query", "message"]`. It keeps the last 200 +`update_id`s that it handled in its state file. An update with one of them +was handled before: it is ignored, a tap and a message alike. A lower ID +that is not in the list still counts, because Telegram may start the IDs +again at a random number after a quiet week. It accepts a tap only when all of these hold, else it answers "This question is closed." and writes nothing: - `from.id` is `owner_id`, and `message.chat.id` is `chat_id`; @@ -372,6 +440,30 @@ these hold, else it answers "This question is closed." and writes nothing: Then it writes the approval, edits the question ("Apply tapped. Waiting for the host." or "Skipped."), and answers the tap. +**`/check`.** The listener writes a check request (section 8) only when all +of these hold, else it writes nothing: + +- `from.id` is `owner_id`, and `chat.id` is `chat_id`; +- it is a new message, not an edited one (no `edited_message`, no + `edit_date`); +- the text is exactly `/check` (a request for every instance in + `instances`) or exactly `/check NAME`, with one space and a NAME that is + in `instances`. Nothing before, after or between: no second word, no new + line, no `@`. So `/check@otherbot` does nothing. The listener does not + know its own bot name, so it refuses `/check@` for its own name too (in a + private chat Telegram sends `/check` without it). + +Then it replies "Checking NAME…" to the message. A second `/check` for the +same instance within 10 seconds writes nothing and gets "a check was asked +a few seconds ago" (section 8, the start limit). When a request cannot be +written (for example the host never ran pullgate for NAME, so the directory +does not exist), it replies that the request was not saved. To the owner in +the chat, a text that starts with `/check` but does not match gets the +usage text, which lists the instances and never repeats the message. A +message of another user, or in another chat, gets no answer. The listener +never logs the text of a message: only why it was ignored and the sender +ID. + **Root** (`pullgate apply NAME`) takes the lock (with a wait). If the pause file exists or `mode = "off"`, it removes the approvals, replies "ignored: the instance is paused" (or off) to each question it knows, and exits. @@ -528,6 +620,12 @@ closed after a cut. 0600). The path unit starts `pullgate apply`, which handles it like an approval from the inbox, then removes it, and reports. The apply needs the same checks (code, time, forward only). For development hosts. + **`/check` on an `auto` instance** starts the run as the timer does, so a + new commit is applied at once, without a tap. This is a choice: an auto + instance applies every new commit within one timer period anyway, so + `/check` only makes it sooner. It cannot apply a commit that the timer + would not apply. The reply says "it is applied now, without a tap". Who + does not want this keeps the instance in `ask` mode. - `off`: `pullgate run` and `pullgate apply` do nothing. `pullgate status NAME` prints the mode, the applied SHA, the open question, @@ -566,7 +664,8 @@ Each rule and each limit has a test. The tests use a fake Telegram server The end-to-end test (`test/e2e`, build tag `e2e`) builds the binary, makes a bare repo with two commits, runs `pullgate run`, the listener against the fake Telegram server, a tap, `pullgate apply`, and checks that the second -commit was applied. +commit was applied. Then it sends `/check` and checks the request, the run +and the reply "No new commit". The tests of `install.sh` (`install_linux_test.go`, at the root) run the script with `/bin/sh` against a fake release: a fake `curl`, `uname`, `gh` diff --git a/README.md b/README.md index 86ae644..bf7e203 100644 --- a/README.md +++ b/README.md @@ -109,10 +109,36 @@ cp contrib/systemd/* /etc/systemd/system/ systemctl daemon-reload systemctl enable --now pullgate-listen.service systemctl enable --now pullgate-apply@NAME.path pullgate-run@NAME.timer +systemctl enable --now pullgate-check@NAME.path # optional: /check ``` Every step logs to the journal: `journalctl -t pullgate`. +## Check now: `/check` + +The timer runs every 5 minutes. After a push you do not have to wait: send +the bot `/check` (all instances) or `/check NAME` (one instance). The +listener answers "Checking NAME…" and writes a check request. +`pullgate-check@NAME.path` then starts the normal run at once. You get the +usual Apply question, or "No new commit (the applied commit is …)", or +"NAME is paused" / "NAME is off" (then nothing is fetched). + +- Only your user ID in your chat counts. The text must be exactly + `/check` or `/check NAME`. Other messages get no request. Edited messages + do not count. +- `/check` never applies anything in `ask` mode: it only fetches and asks. + **In `auto` mode, `/check` applies a new commit at once, without a tap**, + as the timer would a few minutes later. +- Root allows at most one run per minute from `/check`. More requests in + that minute are dropped, with one reply. The listener also takes at most + one `/check` per instance in 10 seconds. +- `pullgate-check@NAME.path` stops after 10 starts in a minute. Then + `/check` waits for the timer until you run + `systemctl reset-failed pullgate-check@NAME.path` and + `systemctl start pullgate-check@NAME.path`. +- Without `pullgate-check@NAME.path`, the request waits for the next timer + run. + ## Config The instance config `/etc/pullgate/NAME.toml`: diff --git a/THREAT-MODEL.md b/THREAT-MODEL.md index 1df3225..df350ce 100644 --- a/THREAT-MODEL.md +++ b/THREAT-MODEL.md @@ -33,7 +33,7 @@ on the host, out of reach of the repo. | the owner's Telegram account and phone | Trusted. Anyone who can use this account can approve. | | Telegram's servers | Trusted to deliver messages and to say truthfully who tapped. They see the question text: SHAs, commit subjects, file names and the summary output. | | root on the host | Fully trusted. | -| the listener process and its user | Trusted only to pass on the owner's taps. If it is taken over, it can approve an open question without the owner (T2). It can never choose the SHA. | +| the listener process and its user | Trusted only to pass on the owner's taps and `/check` messages. If it is taken over, it can approve an open question without the owner (T2), and start runs (at most one a minute, T18). It can never choose the SHA. | | `check_user` and the `check`/`summary` commands | Not trusted with root. They read hostile data (T3). | | the git server and everyone who can push | **Not trusted.** A commit may be hostile. | | the network | Not trusted. SSH with a pinned host key protects the fetch. | @@ -78,6 +78,14 @@ sits above real buttons for a hostile SHA. The listener counts a tap only on a message that was never edited, whose text is the text Telegram gave back when the listener sent it (DESIGN.md section 9). +*`/check`.* A token cannot send a message as the owner either. A message +that the thief sends to the bot, or in another chat, comes with the +thief's own `from.id` (or the wrong `chat.id`); the listener ignores it, +gives no answer and logs no text (DESIGN.md section 9). The thief can read +the owner's `/check` messages, but that tells only that the owner wants a +check. Even a request that the thief could forge would only start a normal +run: it fetches and asks; in `ask` mode it never applies. + *Left.* The thief can show the owner false text (a fake "applied" message), can hide questions, and can see the question text. Keep the token in a file only the listener user can read, or a systemd credential. @@ -99,9 +107,22 @@ the spool directories above them are root-owned, root opens them with random name. So a symlink that the listener plants cannot make root write or change a file elsewhere. +*Check requests.* The listener can also write check requests at any time, +without the owner. A request holds no data: root checks it like an +approval (`O_NOFOLLOW`, regular file, owned by the listener, at most 64 +bytes) and also needs the fixed name and content. It cannot name a SHA, +an instance other than its own directory, or an action. It can only start +the normal run sooner, with the same sandbox, lock and rules. Root keeps +the limit (one run per minute from requests) in its own state, so the +listener cannot make root fetch more often by writing files; a bad file +is not a request (T18). + *Left.* It can approve any open question (or a superseded one that has not expired) without the owner, and it can hide or fake messages. So the attacker still needs a hostile commit in the repo **and** the listener. +With check requests it can make root fetch and ask once a minute, and make +an `auto` instance apply a new commit sooner than the timer would; never a +commit that the timer would not apply. Run the listener as its own user, with the hardened unit. ### T3. A hostile commit @@ -226,7 +247,9 @@ replayed; or an old message is tapped. *Defense.* A lost tap means no apply (fail closed); the question expires and the SHA waits for a new commit or `--ask-again`. The listener forgets a code after the first tap and after the question is closed, so a second tap, or a -tap on a closed or expired question, writes nothing. Root removes a question +tap on a closed or expired question, writes nothing. The listener also +keeps the last 200 update IDs it handled and ignores an update with one of +them, a tap or a `/check` alike. Root removes a question from its state before it acts, so a replayed approval finds no question. Root checks the time limit itself. A tap on a superseded question that the listener took before it handled the close still applies the old SHA @@ -265,6 +288,19 @@ listener as the file owner, a regular file, at most 1 KiB, valid JSON without unknown fields, and a code that matches its state. A bad file is removed and logged. The path unit then has nothing left and stops. +*The check directory.* A check request has no code, so a code cannot +protect it. It needs none, because it cannot cause an apply: it only +starts the normal run, which asks (or, in `auto` mode, does what the timer +does). Only the listener user can write the directory. Root needs the +listener as owner, a regular file through `O_NOFOLLOW`, at most 64 bytes, +the name `request` and the content `check\n`. Root removes every file in +the directory before it acts, and a bad file is not a request. +`pullgate apply` never reads this directory, and a check request put in +the inbox is a bad approval. A run that finds only bad files removes them +and goes on as a normal run, without a reply. The limit of T18 holds for +every valid request; the trigger limit of the path unit holds for every +file. + ### T13. A hostile git server or network *Attack.* A fake server, or a server that sends huge or broken data. @@ -359,6 +395,31 @@ through a proxy. The old binary runs once, as root, to print its version; only root can change it, so this gives nothing new to an attacker. The script restarts nothing. +### T18. Many check requests + +*Attack.* The owner, by mistake, or a taken-over listener sends many +`/check` requests, so the host fetches all the time and loads the git +server. + +*Defense.* Root allows at most one run per minute from the check directory. +It keeps the time in its own state, not in the listener. A run that finds a +request inside that minute removes it, does not fetch, and sends at most one +reply per minute. A clock that goes back by less than a minute does not +pass the limit. Many `/check` messages before root reads them are one +file (a fixed name). The path unit starts the same run unit as the timer, +and systemd never starts it twice at once; the lock keeps a run and an +apply apart. A run that sees a request while the lock is held waits for +it, instead of starting again and again. The listener writes at most one +request per instance in 10 seconds, so a burst of `/check` from the owner +does not reach the trigger limit of the path unit. + +*Left.* One fetch per minute, and one reply per minute. A taken-over +listener can write requests fast enough to hit the trigger limit of the +path unit; systemd then stops the path unit, and `/check` does nothing +until the owner resets it (DESIGN.md section 8). The run unit has no start +limit, so the timer goes on. This denies; it never approves. A clock jump back of more than +a minute allows one more run. + ## 6. Out of scope - An attacker who is root on the host. diff --git a/contrib/systemd/pullgate-check@.path b/contrib/systemd/pullgate-check@.path new file mode 100644 index 0000000..cf6567e --- /dev/null +++ b/contrib/systemd/pullgate-check@.path @@ -0,0 +1,23 @@ +# Starts pullgate-run@NAME.service when the listener writes a check request +# for /check on Telegram (DESIGN.md section 8). The run is the normal run: +# the same unit, sandbox and lock. Root removes the request and allows at +# most one requested run per minute. +[Unit] +Description=pullgate check requests for %i + +[Path] +# Only the request file counts, so another file in the directory does not +# start runs again and again. +PathExists=/var/spool/pullgate/%i/check/request +Unit=pullgate-run@%i.service +# At most 10 starts a minute. This stops a loop when a run fails before it +# removes the request. The listener writes at most one request per instance +# in 10 seconds, so /check alone does not reach it. When it is reached, +# this unit stops; the timer goes on. Start it again with: +# systemctl reset-failed pullgate-check@NAME.path +# systemctl start pullgate-check@NAME.path +TriggerLimitIntervalSec=60s +TriggerLimitBurst=10 + +[Install] +WantedBy=paths.target diff --git a/contrib/systemd/pullgate-run@.service b/contrib/systemd/pullgate-run@.service index 73f488a..48c1344 100644 --- a/contrib/systemd/pullgate-run@.service +++ b/contrib/systemd/pullgate-run@.service @@ -1,10 +1,14 @@ # pullgate run NAME: fetch, check, and ask (DESIGN.md section 7). -# Started by pullgate-run@NAME.timer. Runs as root, because root owns the -# clone (rule 6). check and summary run as check_user inside this sandbox. +# Started by pullgate-run@NAME.timer, and by pullgate-check@NAME.path on +# /check. Runs as root, because root owns the clone (rule 6). check and +# summary run as check_user inside this sandbox. [Unit] Description=pullgate run for %i After=network-online.target Wants=network-online.target +# No start limit here: pullgate-check@NAME.path has the limit of /check +# (TriggerLimitBurst=), so many check requests can never stop the timer. +StartLimitIntervalSec=0 [Service] Type=oneshot diff --git a/install.sh b/install.sh index 9bd0c87..d01591e 100644 --- a/install.sh +++ b/install.sh @@ -681,7 +681,7 @@ print_restart() { fi say "This script restarts nothing. The listener still runs the old version. Restart it:" say " systemctl restart pullgate-listen.service" - say "The timers (pullgate-run@NAME.timer) and the path units (pullgate-apply@NAME.path)" + say "The timers (pullgate-run@NAME.timer) and the path units (pullgate-apply@NAME.path, pullgate-check@NAME.path)" say "start a new process each time, so they use the new file from their next start." say "A run or an apply that is running now ends with the old version. See them with:" say " systemctl list-timers 'pullgate-run@*'" diff --git a/install_linux_test.go b/install_linux_test.go index 7ee3975..65687a1 100644 --- a/install_linux_test.go +++ b/install_linux_test.go @@ -539,7 +539,7 @@ func TestInstall(t *testing.T) { in.checkTmpGone() for _, s := range []string{"Installing pullgate 0.2.0 as " + in.bin + "/pullgate.", "version says: pullgate 0.2.0", "This script restarts nothing.", "systemctl restart pullgate-listen.service", "pullgate-run@NAME.timer", - "pullgate-apply@NAME.path"} { + "pullgate-apply@NAME.path", "pullgate-check@NAME.path"} { if !strings.Contains(r.stdout, s) { t.Errorf("stdout does not say %q:\n%s", s, r.stdout) } diff --git a/internal/gate/check.go b/internal/gate/check.go new file mode 100644 index 0000000..9ed4579 --- /dev/null +++ b/internal/gate/check.go @@ -0,0 +1,87 @@ +package gate + +import ( + "time" + + "github.com/code6x/pullgate/internal/state" +) + +// CheckInterval is the least time between two runs that check requests +// start (DESIGN.md section 8). Root keeps the time in its own state, so a +// listener that writes many requests cannot make the host fetch more often. +const CheckInterval = time.Minute + +// takeChecks reads and removes the files in the check directory, and +// returns how many were valid check requests. A bad file is logged and +// removed. An error of the directory counts as no request (fail closed). +func (e *Env) takeChecks() int { + n, bad, err := e.Spool.TakeChecks(e.Owners.ListenerUID) + for _, b := range bad { + e.Log.Warn("bad check request, removed", "err", b) + } + if err != nil { + e.Log.Error("read the check requests", "err", err) + } + if n > 0 { + e.Log.Info("check request taken") + } + return n +} + +// limitCheck enforces CheckInterval. It returns CheckLimited when the +// request must be dropped; else it records the time of this request. The +// drop gets one reply in each interval. +func (e *Env) limitCheck(st *state.State) Result { + now := e.now() + since := now.Sub(st.CheckAt) + if since < CheckInterval && since > -CheckInterval { + e.Log.Warn("check request dropped: the last one was less than a minute ago") + if d := now.Sub(st.CheckNote); d >= CheckInterval || d <= -CheckInterval { + st.CheckNote = now + e.notice("A check ran less than a minute ago, so this /check was dropped. Try again in a minute.") + } + return CheckLimited + } + st.CheckAt = now + return "" +} + +// checkReply tells the owner what a run that a check request started found, +// when no other message says it already. limitSent is true when the run +// sent the questions_per_day notice. +func (e *Env) checkReply(st *state.State, res Result, err error, limitSent bool) { + if err != nil { + e.notice("The check failed. See the journal of the host: journalctl -u pullgate-run@" + e.Cfg.Name + ".service") + return + } + tip := st.LastTip + switch res { + case Nothing: + switch open := st.Open(); { + case tip == st.Applied: + e.notice("No new commit. The applied commit is " + code(tip) + ".") + case open != nil && open.SHA == tip && !open.Auto: + e.notice("No new commit. The question for " + code(tip) + " is still open.") + case open != nil && open.SHA == tip: + e.notice("No new commit. " + code(tip) + " is waiting for its auto apply.") + case tip == st.Held: + e.notice("No new question: the last apply of " + code(tip) + " failed and it is held. A newer commit is asked.") + case st.Marks[tip] != nil && st.Marks[tip].Mark == state.MarkRefused: + e.notice("No new question: " + code(tip) + " is not a descendant of the applied commit " + code(st.Applied) + ", so it was refused. To allow it, set allow_older = true in [limits] of the host config.") + default: + mark := "" + if m := st.Marks[tip]; m != nil { + mark = " (" + m.Mark + ")" + } + e.notice("No new question: " + code(tip) + " was handled before" + mark + ". To ask again, run as root: pullgate status " + e.Cfg.Name + " --ask-again") + } + case AutoQueued: + e.notice("New commit " + code(tip) + ". The mode is auto (mode = \"auto\"), so it is applied now, without a tap.") + case LimitReached: + if !limitSent { + e.notice("No new question for " + code(tip) + ": the limit questions_per_day is reached.") + } + case Asked, Refused, CheckFailed: + // The question or a notice says it already. + } +} diff --git a/internal/gate/check_test.go b/internal/gate/check_test.go new file mode 100644 index 0000000..56132bd --- /dev/null +++ b/internal/gate/check_test.go @@ -0,0 +1,333 @@ +package gate + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/code6x/pullgate/internal/spool" +) + +// request writes a check request, as the listener does on /check. +func (f *fixture) request() { + f.t.Helper() + if err := f.env.Spool.Ensure(f.env.UID, f.env.Owners); err != nil { + f.t.Fatal(err) + } + if err := f.env.Spool.WriteCheck(); err != nil { + f.t.Fatal(err) + } +} + +func TestCheckRequestStartsANormalRun(t *testing.T) { + f := newFixture(t, ``, "") + f.commit("first", "one", nil) + f.request() + if r := f.run(); r != Asked { + t.Fatalf("run: %s, want asked", r) + } + q := f.ask() + if f.env.Spool.HasCheck() { + t.Fatal("root must remove the request") + } + // No new commit: a reply says so. + f.now = f.now.Add(2 * time.Minute) + f.request() + if r := f.run(); r != Nothing { + t.Fatalf("run: %s", r) + } + if !contains(f.outbox(), "is still open") { + t.Fatal("the reply must say that the question is still open") + } + f.tap(q.Code, "apply") + if r := f.apply(); r != Applied { + t.Fatalf("apply: %s", r) + } + f.outbox() + f.now = f.now.Add(2 * time.Minute) + f.request() + if r := f.run(); r != Nothing { + t.Fatalf("run: %s", r) + } + if !contains(f.outbox(), "No new commit. The applied commit is "+f.state().Applied+"") { + t.Fatal("the reply must name the applied SHA") + } +} + +func TestCheckRequestNeverAppliesWithoutTap(t *testing.T) { + f := newFixture(t, ``, "") + f.commit("first", "one", nil) + for i := range 3 { + f.request() + f.run() + f.apply() + f.now = f.now.Add(time.Duration(i+2) * time.Minute) + } + if st := f.state(); st.Applied != "" || st.Open() == nil { + t.Fatalf("mode ask: a check request must only ask, never apply: %+v", st) + } + if f.read("log") != "" { + t.Fatal("the apply command must not run without a tap") + } + // A check request put in the inbox is a bad approval, not a tap. + if err := os.WriteFile(filepath.Join(f.env.Spool.Inbox(), "1.json"), []byte(spool.CheckContent), 0o600); err != nil { + t.Fatal(err) + } + if r := f.apply(); r == Applied || f.state().Applied != "" { + t.Fatalf("apply: %s", r) + } +} + +func TestCheckRateLimitByRoot(t *testing.T) { + f := newFixture(t, ``, "") + a := f.commit("first", "one", nil) + f.request() + if r := f.run(); r != Asked { + t.Fatalf("run: %s", r) + } + f.outbox() + b := f.commit("second", "two", nil) + f.now = f.now.Add(30 * time.Second) + f.request() + if r := f.run(); r != CheckLimited { + t.Fatalf("a second request in a minute must be dropped: %s", r) + } + if f.state().LastTip != a { + t.Fatal("a dropped request must not fetch") + } + if ms := f.outbox(); len(ms) != 1 || !contains(ms, "was dropped") { + t.Fatalf("one reply for the drop: %+v", ms) + } + f.now = f.now.Add(10 * time.Second) + f.request() + if r := f.run(); r != CheckLimited { + t.Fatalf("run: %s", r) + } + if len(f.outbox()) != 0 { + t.Fatal("only one reply for drops in a minute") + } + // A clock that goes back does not pass the limit. + f.now = f.now.Add(-50 * time.Second) + f.request() + if r := f.run(); r != CheckLimited { + t.Fatalf("run: %s", r) + } + f.now = f.now.Add(2 * time.Minute) + f.request() + if r := f.run(); r != Asked || f.state().LastTip != b { + t.Fatalf("a minute later the request runs: %s", r) + } +} + +// TestCheckBadRequestFiles: a file of another owner needs root to make; +// spool.TestBadCheckRequests covers it ("wrong owner"). +func TestCheckBadRequestFiles(t *testing.T) { + cases := map[string]func(f *fixture){ + "symlink": func(f *fixture) { + if err := os.Symlink(f.env.Dir.StateFile(), filepath.Join(f.env.Spool.CheckDir(), spool.CheckFile)); err != nil { + f.t.Fatal(err) + } + }, + "too big": func(f *fixture) { + if err := os.WriteFile(filepath.Join(f.env.Spool.CheckDir(), spool.CheckFile), make([]byte, 100), 0o600); err != nil { + f.t.Fatal(err) + } + }, + } + for n, c := range cases { + t.Run(n, func(t *testing.T) { + f := newFixture(t, ``, "") + f.commit("first", "one", nil) + if err := f.env.Spool.Ensure(f.env.UID, f.env.Owners); err != nil { + t.Fatal(err) + } + c(f) + // A timer run that finds only bad files removes them and + // goes on as a normal run, with no reply. + if r := f.run(); r != Asked { + t.Fatalf("the run must go on as a normal run: %s", r) + } + if left, _ := os.ReadDir(f.env.Spool.CheckDir()); len(left) != 0 { + t.Fatal("a bad request must be removed") + } + if st := f.state(); !st.CheckAt.IsZero() { + t.Fatal("a bad request must not count as a request") + } + for _, m := range f.outbox() { + if m.Kind != spool.KindAsk { + t.Fatalf("a bad request gets no reply: %+v", m) + } + } + }) + } +} + +func TestCheckPausedAndOff(t *testing.T) { + for _, mode := range []string{"paused", "off"} { + t.Run(mode, func(t *testing.T) { + top := `` + if mode == "off" { + top = `mode = "off"` + } + f := newFixture(t, top, "") + f.commit("first", "one", nil) + if mode == "paused" { + if err := f.env.Dir.Ensure(); err != nil { + t.Fatal(err) + } + if err := f.env.Dir.Pause("test"); err != nil { + t.Fatal(err) + } + } + f.request() + r := f.run() + if string(r) != mode { + t.Fatalf("run: %s", r) + } + if !contains(f.outbox(), "nothing was fetched") { + t.Fatal("the reply must say so") + } + if st := f.state(); st.LastTip != "" || len(st.Questions) != 0 { + t.Fatalf("a paused or off instance must not fetch: %+v", st) + } + if f.env.Spool.HasCheck() { + t.Fatal("the request must be removed") + } + }) + } +} + +func TestCheckWaitsForTheLock(t *testing.T) { + f := newFixture(t, ``, "") + f.commit("first", "one", nil) + if err := f.env.Dir.Ensure(); err != nil { + t.Fatal(err) + } + l, err := f.env.Dir.Lock(false) + if err != nil { + t.Fatal(err) + } + f.request() + done := make(chan Result, 1) + go func() { + r, err := f.env.Run(context.Background()) + if err != nil { + r = Result("error: " + err.Error()) + } + done <- r + }() + select { + case r := <-done: + t.Fatalf("the run must wait while the lock is held: %s", r) + case <-time.After(300 * time.Millisecond): + } + if st := f.state(); st.LastTip != "" { + t.Fatal("nothing may run while the lock is held") + } + _ = l.Close() + if r := <-done; r != Asked { + t.Fatalf("after the lock is free the request runs once: %s", r) + } +} + +func TestCheckAutoModeApplies(t *testing.T) { + // Decision (DESIGN.md section 13): /check on an auto instance runs as + // the timer does, so a new commit is applied without a tap. + f := newFixture(t, `mode = "auto"`, "") + sha := f.commit("first", "one", nil) + f.request() + if r := f.run(); r != AutoQueued { + t.Fatalf("run: %s", r) + } + if !contains(f.outbox(), "applied now, without a tap") { + t.Fatal("the reply must say that auto mode applies it") + } + if r := f.apply(); r != Applied || f.state().Applied != sha { + t.Fatalf("apply: %s", r) + } +} + +func TestCheckReplyForRefusedTip(t *testing.T) { + f := newFixture(t, ``, "") + f.applyNew("one") + f.r.Git("checkout", "--quiet", "--orphan", "other") + f.commit("rewritten", "evil", nil) // not a descendant + if r := f.run(); r != Refused { + t.Fatalf("run: %s", r) + } + f.outbox() + f.now = f.now.Add(2 * time.Minute) + f.request() + if r := f.run(); r != Nothing { + t.Fatalf("run: %s", r) + } + if ms := f.outbox(); !contains(ms, "allow_older = true") || contains(ms, "--ask-again") { + t.Fatalf("the reply must point to allow_older: %+v", ms) + } +} + +func TestCheckLimitReplySentOnce(t *testing.T) { + f := newFixture(t, ``, "[limits]\nquestions_per_day = 1\n") + f.commit("first", "one", nil) + f.run() + f.outbox() + f.commit("second", "two", nil) + f.request() + if r := f.run(); r != LimitReached { + t.Fatalf("run: %s", r) + } + n := 0 + for _, m := range f.outbox() { + if strings.Contains(m.Text, "questions_per_day") { + n++ + } + } + if n != 1 { + t.Fatalf("one message about the limit, got %d", n) + } +} + +// TestCheckWaiterStopsWhenRequestIsGone: a run that waited for the lock and +// then finds no request (another run took it) stops: no fetch, no reply. +func TestCheckWaiterStopsWhenRequestIsGone(t *testing.T) { + f := newFixture(t, ``, "") + f.commit("first", "one", nil) + if err := f.env.Dir.Ensure(); err != nil { + t.Fatal(err) + } + l, err := f.env.Dir.Lock(false) + if err != nil { + t.Fatal(err) + } + f.request() + done := make(chan Result, 1) + go func() { + r, err := f.env.Run(context.Background()) + if err != nil { + r = Result("error: " + err.Error()) + } + done <- r + }() + select { + case r := <-done: + t.Fatalf("the run must wait while the lock is held: %s", r) + case <-time.After(300 * time.Millisecond): + } + // The run that holds the lock takes the request. + if n, _, err := f.env.Spool.TakeChecks(os.Getuid()); err != nil || n != 1 { + t.Fatalf("TakeChecks: %d, %v", n, err) + } + _ = l.Close() + if r := <-done; r != Busy { + t.Fatalf("a waiter without a request must stop as busy: %s", r) + } + if st := f.state(); st.LastTip != "" { + t.Fatal("the waiter must not fetch") + } + if out := f.outbox(); len(out) != 0 { + t.Fatalf("the waiter must not reply: %d messages", len(out)) + } +} diff --git a/internal/gate/run.go b/internal/gate/run.go index bc17508..345470d 100644 --- a/internal/gate/run.go +++ b/internal/gate/run.go @@ -26,6 +26,7 @@ const ( Nothing Result = "nothing" Refused Result = "refused" LimitReached Result = "limit" + CheckLimited Result = "check-limited" CheckFailed Result = "check-failed" Asked Result = "asked" AutoQueued Result = "auto" @@ -108,38 +109,89 @@ func (e *Env) closeOpen(st *state.State, why string) { } } -// Run is one "pullgate run" (DESIGN.md section 7). +// Run is one "pullgate run" (DESIGN.md section 7). It also takes the check +// requests of /check (DESIGN.md section 8). func (e *Env) Run(ctx context.Context) (Result, error) { if err := e.Dir.Ensure(); err != nil { return "", err } lock, err := e.Dir.Lock(false) if errors.Is(err, state.ErrBusy) { - e.Log.Info("busy: another run or apply holds the lock") - return Busy, nil + if !e.Spool.HasCheck() { + e.Log.Info("busy: another run or apply holds the lock") + return Busy, nil + } + // A check request waits for the lock, so that it is not lost and + // the path unit does not start the run again and again. + e.Log.Info("busy: a check request waits for the lock") + lock, err = e.Dir.Lock(true) + if err == nil && !e.Spool.HasCheck() { + // Another run took the request (and replied, if it was a good one). + _ = lock.Close() + e.Log.Info("busy: the run before this one took the check request") + return Busy, nil + } } if err != nil { return "", err } defer func() { _ = lock.Close() }() - if e.Dir.Paused() { - e.Log.Info("paused: the pause file exists", "file", e.Dir.PauseFile()) - return Paused, nil - } - if e.Cfg.Mode == config.ModeOff { + paused, off := e.Dir.Paused(), e.Cfg.Mode == config.ModeOff + if (paused || off) && !e.Spool.HasCheck() { + if paused { + e.Log.Info("paused: the pause file exists", "file", e.Dir.PauseFile()) + return Paused, nil + } e.Log.Info("off: mode is off") return Off, nil } if err := e.Spool.Ensure(e.UID, e.Owners); err != nil { return "", err } + // A bad file is removed and is not a request: the run goes on as a + // normal run, with no reply and no mark against the limit. + requests := e.takeChecks() st, err := e.Dir.Load() if err != nil { return "", err } + requested := requests > 0 + if requested { + if r := e.limitCheck(st); r != "" { + return r, e.Dir.Save(st) + } + if err := e.Dir.Save(st); err != nil { + return "", err + } + } + if paused || off { + why := e.Cfg.Name + " is off (mode = \"off\"), so nothing was fetched." + res := Off + if paused { + why = e.Cfg.Name + " is paused, so nothing was fetched. Run as root: pullgate status " + e.Cfg.Name + " --resume" + res = Paused + } + e.Log.Info(string(res)) + if requested { + e.notice(why) + } + return res, nil + } if st.InProgress != nil { return CutOff, e.cutOff(st) } + limitNote := st.LimitNote + res, err := e.fetchAndDecide(ctx, st) + if requested { + e.checkReply(st, res, err, !st.LimitNote.Equal(limitNote)) + } + if serr := e.Dir.Save(st); serr != nil && err == nil { + err = serr + } + return res, err +} + +func (e *Env) fetchAndDecide(ctx context.Context, st *state.State) (Result, error) { if err := e.Git.Init(ctx); err != nil { return "", err } @@ -151,11 +203,7 @@ func (e *Env) Run(ctx context.Context) (Result, error) { e.Log.Info("fetched", "tip", tip, "applied", st.Applied) st.LastTip = tip e.expire(st) - res, err := e.decide(ctx, st, tip) - if serr := e.Dir.Save(st); serr != nil && err == nil { - err = serr - } - return res, err + return e.decide(ctx, st, tip) } func (e *Env) decide(ctx context.Context, st *state.State, tip string) (Result, error) { diff --git a/internal/listener/listener.go b/internal/listener/listener.go index 58d6fb0..5fe01a0 100644 --- a/internal/listener/listener.go +++ b/internal/listener/listener.go @@ -17,6 +17,7 @@ import ( "io/fs" "log/slog" "os" + "slices" "strings" "sync" "time" @@ -44,9 +45,20 @@ type Question struct { // State is the listener's state file. type State struct { Offset int64 `json:"offset"` + Seen []int64 `json:"seen,omitempty"` // the last update IDs handled, at most MaxSeen Questions []Question `json:"questions"` } +// MaxSeen is how many handled update IDs the listener keeps, so that a +// replayed update is ignored. +const MaxSeen = 200 + +// CheckGap is the least time between two check requests that the listener +// writes for one instance. Root keeps its own limit (DESIGN.md section 8); +// this one keeps a burst of /check from hitting the start limit of the +// run unit. +const CheckGap = 10 * time.Second + // Listener holds the parts. type Listener struct { Cfg *config.Listener @@ -56,8 +68,9 @@ type Listener struct { Poll time.Duration // how often the outboxes are read LongPoll int // getUpdates timeout in seconds - mu sync.Mutex - st State + mu sync.Mutex + st State + lastCheck map[string]time.Time // in memory: the last check request per instance } func hash(s string) string { @@ -263,22 +276,40 @@ func (l *Listener) send(ctx context.Context, inst string, m spool.Message) error } } -// Update handles one update: only a valid tap of the owner counts. +// Update handles one update: only a valid tap or /check of the owner +// counts. An update with an ID below the offset was handled before and is +// ignored. func (l *Listener) Update(ctx context.Context, u telegram.Update) { l.mu.Lock() defer l.mu.Unlock() - if u.UpdateID >= l.st.Offset { - l.st.Offset = u.UpdateID + 1 + if slices.Contains(l.st.Seen, u.UpdateID) { + l.Log.Info("update ignored", "reason", "an update that was handled before", "update", u.UpdateID) + return } + // Telegram may start the IDs again at a random number after a quiet + // week, so a lower ID is not dropped; only an ID that was handled. + l.st.Seen = append(l.st.Seen, u.UpdateID) + if len(l.st.Seen) > MaxSeen { + l.st.Seen = l.st.Seen[len(l.st.Seen)-MaxSeen:] + } + l.st.Offset = u.UpdateID + 1 defer func() { if err := l.save(); err != nil { l.Log.Error("save the listener state", "err", err) } }() - cq := u.CallbackQuery - if cq == nil { - return + switch { + case u.CallbackQuery != nil: + l.tap(ctx, u.CallbackQuery) + case u.Message != nil: + l.command(ctx, u.Message) + case u.EditedMessage != nil: + l.Log.Info("message ignored", "reason", "an edited message") } +} + +// tap handles a button tap. +func (l *Listener) tap(ctx context.Context, cq *telegram.CallbackQuery) { answer := Closed defer func() { if err := l.TG.AnswerCallbackQuery(ctx, cq.ID, answer); err != nil { @@ -349,3 +380,84 @@ func (l *Listener) check(cq *telegram.CallbackQuery) (*Question, string, string) } return q, action, "" } + +// checkCommand is the one command the listener takes (DESIGN.md section 9). +const checkCommand = "/check" + +// command handles a message. Only "/check" or "/check NAME" of the owner, +// in the chat, unedited, counts. The text of a message is never logged. +func (l *Listener) command(ctx context.Context, m *telegram.Message) { + names, why, help := l.checkNames(m) + if names == nil { + l.Log.Info("message ignored", "reason", why, "from", fromID(m)) + if help { + l.sendReply(ctx, m, "Use /check for all instances, or /check NAME. NAME is one of: "+html.EscapeString(strings.Join(l.Cfg.Instances, ", "))+".") + } + return + } + var ok, failed, soon []string + if l.lastCheck == nil { + l.lastCheck = map[string]time.Time{} + } + now := l.now() + for _, n := range names { + if d := now.Sub(l.lastCheck[n]); d < CheckGap && d > -CheckGap { + soon = append(soon, n) + continue + } + if err := l.dirs(n).WriteCheck(); err != nil { + l.Log.Error("write the check request", "instance", n, "err", err) + failed = append(failed, n) + continue + } + l.lastCheck[n] = now + ok = append(ok, n) + } + l.Log.Info("check request taken", "instances", ok) + var text []string + if len(ok) > 0 { + text = append(text, "Checking "+html.EscapeString(strings.Join(ok, ", "))+"…") + } + if len(soon) > 0 { + text = append(text, ""+html.EscapeString(strings.Join(soon, ", "))+": a check was asked a few seconds ago. Wait for its answer.") + } + if len(failed) > 0 { + text = append(text, "Error: the check request for "+html.EscapeString(strings.Join(failed, ", "))+" was not saved. Check the journal of the listener, and that the host ran pullgate once for it.") + } + l.sendReply(ctx, m, strings.Join(text, "\n")) +} + +// checkNames returns the instances that a message asks to check, or nil, +// why not, and whether the owner should get the usage text. +func (l *Listener) checkNames(m *telegram.Message) ([]string, string, bool) { + switch { + case m.From == nil || m.From.ID != l.Cfg.OwnerID: + return nil, "not the owner", false + case m.Chat.ID != l.Cfg.ChatID: + return nil, "not the configured chat", false + case m.EditDate != 0: + return nil, "an edited message", false + case !strings.HasPrefix(m.Text, checkCommand): + return nil, "not a command", false + case m.Text == checkCommand: + return slices.Clone(l.Cfg.Instances), "", false + } + name, ok := strings.CutPrefix(m.Text, checkCommand+" ") + if !ok || !slices.Contains(l.Cfg.Instances, name) { + return nil, "not /check or /check NAME of a known instance", true + } + return []string{name}, "", false +} + +func fromID(m *telegram.Message) int64 { + if m.From == nil { + return 0 + } + return m.From.ID +} + +func (l *Listener) sendReply(ctx context.Context, m *telegram.Message, text string) { + if _, err := l.TG.SendMessage(ctx, l.Cfg.ChatID, text, nil, m.MessageID); err != nil { + l.Log.Warn("send a reply", "err", err) + } +} diff --git a/internal/listener/listener_test.go b/internal/listener/listener_test.go index 9f83287..9df4846 100644 --- a/internal/listener/listener_test.go +++ b/internal/listener/listener_test.go @@ -211,3 +211,167 @@ func TestTokenNeverInErrors(t *testing.T) { t.Fatalf("the error must not hold the token: %v", err) } } + +const inst2 = "playbooks" + +// newL2 is newL with two instances. +func newL2(t *testing.T) (*lfix, spool.Dirs) { + f := newL(t) + d2 := spool.Dirs{Base: f.d.Base, Name: inst2} + if err := d2.Ensure(uint32(os.Geteuid()), spool.Owners{ListenerUID: os.Geteuid(), ListenerGID: os.Getegid()}); err != nil { + t.Fatal(err) + } + f.l.Cfg.Instances = append(f.l.Cfg.Instances, inst2) + return f, d2 +} + +// say sends one message through getUpdates. +func (f *lfix) say(from, chatID int64, text string, edited bool) { + f.t.Helper() + f.tg.Say(from, chatID, text, edited) + ups, err := f.l.TG.GetUpdates(context.Background(), f.l.st.Offset, 0) + if err != nil { + f.t.Fatal(err) + } + for _, u := range ups { + f.l.Update(context.Background(), u) + } +} + +func TestCheckCommandWritesRequests(t *testing.T) { + f, d2 := newL2(t) + f.say(owner, chat, "/check "+inst2, false) + if f.d.HasCheck() || !d2.HasCheck() { + t.Fatal("/check NAME must write a request for that instance only") + } + ms := f.tg.Messages() + if len(ms) != 1 || !strings.Contains(ms[0].Text, "Checking "+inst2+"") || ms[0].ReplyTo == 0 { + t.Fatalf("the reply must say checking: %+v", ms) + } + if n, bad, err := d2.TakeChecks(os.Geteuid()); n != 1 || len(bad) != 0 || err != nil { + t.Fatalf("the request must be valid for root: %d %v %v", n, bad, err) + } + f.now = f.now.Add(CheckGap) + f.say(owner, chat, "/check", false) + if !f.d.HasCheck() || !d2.HasCheck() { + t.Fatal("/check must write a request for every instance") + } +} + +func TestCheckCommandsThatDoNotCount(t *testing.T) { + cases := map[string]struct { + from, chat int64 + text string + edited bool + help bool // the owner gets the usage text + }{ + "another user": {owner + 1, chat, "/check", false, false}, + "another chat": {owner, chat + 1, "/check", false, false}, + "edited message": {owner, chat, "/check", true, false}, + "unknown name": {owner, chat, "/check other", false, true}, + "extra text": {owner, chat, "/check " + inst + " now", false, true}, + "two spaces": {owner, chat, "/check " + inst, false, true}, + "to another bot": {owner, chat, "/check@otherbot", false, true}, + "name with @": {owner, chat, "/check@otherbot " + inst, false, true}, + "other text": {owner, chat, "hello", false, false}, + "command in a line": {owner, chat, "please /check", false, false}, + "other user, known": {owner + 1, chat, "/check " + inst, false, false}, + "trailing new line": {owner, chat, "/check\n", false, true}, + "upper case command": {owner, chat, "/CHECK", false, false}, + } + for n, c := range cases { + t.Run(n, func(t *testing.T) { + f, d2 := newL2(t) + f.say(c.from, c.chat, c.text, c.edited) + if f.d.HasCheck() || d2.HasCheck() { + t.Fatal("no request may be written") + } + ms := f.tg.Messages() + switch { + case c.help && (len(ms) != 1 || ms[0].Chat != chat || !strings.Contains(ms[0].Text, "Use /check")): + t.Fatalf("the owner must get the usage text: %+v", ms) + case !c.help && len(ms) != 0: + t.Fatalf("no answer may be sent: %+v", ms) + } + for _, m := range ms { + if strings.Contains(m.Text, "other") || strings.Contains(m.Text, "now") { + t.Fatalf("the reply must not repeat the text of the message: %q", m.Text) + } + } + }) + } +} + +func TestReplayedUpdateIsIgnored(t *testing.T) { + f := newL(t) + f.say(owner, chat, "/check", false) + if _, _, err := f.d.TakeChecks(os.Geteuid()); err != nil { + t.Fatal(err) + } + f.now = f.now.Add(time.Minute) // past CheckGap + u := telegram.Update{UpdateID: f.l.st.Offset - 1, Message: &telegram.Message{MessageID: 1, From: &telegram.User{ID: owner}, Chat: telegram.Chat{ID: chat}, Text: "/check"}} + f.l.Update(context.Background(), u) + if f.d.HasCheck() { + t.Fatal("a replayed update_id must do nothing") + } + // A replayed tap too, even if the code were still open. + msg := f.ask(1, code) + f.tap(owner, chat, msg, "a:"+code) + f.approvals() + old := f.l.st.Offset - 1 + f.find(inst, 1).Code = code + f.l.Update(context.Background(), telegram.Update{UpdateID: old, CallbackQuery: &telegram.CallbackQuery{ID: "cq", From: telegram.User{ID: owner}, Message: &telegram.Message{MessageID: msg, Chat: telegram.Chat{ID: chat}}, Data: "a:" + code}}) + if es := f.approvals(); len(es) != 0 { + t.Fatal("a replayed tap must do nothing") + } + // A new, lower ID (Telegram starts again after a quiet week) counts. + f.l.Update(context.Background(), telegram.Update{UpdateID: 1000000 + 7, Message: &telegram.Message{MessageID: 2, From: &telegram.User{ID: owner}, Chat: telegram.Chat{ID: chat}, Text: "/check"}}) + _, _, _ = f.d.TakeChecks(os.Geteuid()) + f.now = f.now.Add(time.Minute) + f.l.Update(context.Background(), telegram.Update{UpdateID: 5, Message: &telegram.Message{MessageID: 3, From: &telegram.User{ID: owner}, Chat: telegram.Chat{ID: chat}, Text: "/check"}}) + if !f.d.HasCheck() || f.l.st.Offset != 6 { + t.Fatalf("a new lower update_id must count, offset %d", f.l.st.Offset) + } +} + +func TestCheckGapInTheListener(t *testing.T) { + f := newL(t) + f.say(owner, chat, "/check", false) + _, _, _ = f.d.TakeChecks(os.Geteuid()) + f.now = f.now.Add(CheckGap / 2) + f.say(owner, chat, "/check "+inst, false) + if f.d.HasCheck() { + t.Fatal("a second /check within CheckGap must write nothing") + } + if ms := f.tg.Messages(); !strings.Contains(ms[len(ms)-1].Text, "a few seconds ago") { + t.Fatalf("the owner must learn why: %+v", ms[len(ms)-1]) + } + f.now = f.now.Add(CheckGap) + f.say(owner, chat, "/check", false) + if !f.d.HasCheck() { + t.Fatal("after CheckGap /check works again") + } +} + +func (f *lfix) find(i string, id int64) *Question { return f.l.find(i, id) } + +func TestCheckRequestWriteFailsSaysSo(t *testing.T) { + f := newL(t) + if err := os.RemoveAll(f.d.CheckDir()); err != nil { + t.Fatal(err) + } + f.say(owner, chat, "/check", false) + ms := f.tg.Messages() + if len(ms) != 1 || !strings.Contains(ms[0].Text, "was not saved") { + t.Fatalf("the owner must learn that the request was not saved: %+v", ms) + } +} + +func TestListenerAsksForMessages(t *testing.T) { + // getUpdates must ask for messages, or /check never arrives. + f := newL(t) + f.say(owner, chat, "/check", false) + if !f.d.HasCheck() { + t.Fatal("the message must arrive") + } +} diff --git a/internal/spool/spool.go b/internal/spool/spool.go index 255bf3b..43e71e8 100644 --- a/internal/spool/spool.go +++ b/internal/spool/spool.go @@ -1,6 +1,6 @@ // Package spool holds the files between root and the listener (DESIGN.md -// section 8): the outbox (root to listener) and the inbox (listener to -// root). Root never follows a path that the listener can change: it works +// section 8): the outbox (root to listener), the inbox (listener to root) +// and the check directory (check requests, listener to root). Root never follows a path that the listener can change: it works // through os.Root, and creates files with O_EXCL|O_NOFOLLOW. package spool @@ -73,6 +73,19 @@ func (d Dirs) Outbox() string { return filepath.Join(d.inst(), "outbox") } // Inbox is the path of the inbox. func (d Dirs) Inbox() string { return filepath.Join(d.inst(), "inbox") } +// CheckDir is the path of the directory of check requests. +func (d Dirs) CheckDir() string { return filepath.Join(d.inst(), "check") } + +// CheckFile is the one name a check request has in the check directory. +const CheckFile = "request" + +// CheckContent is the whole content of a check request. It holds no data +// from the message (DESIGN.md section 8). +const CheckContent = "check\n" + +// MaxCheck is the largest check request file root reads. +const MaxCheck = 64 + // Ensure makes the spool of the instance with the owners and modes of // DESIGN.md section 4. Base and Base/NAME belong to root (uid) and must // pass the check of rule 6; outbox and inbox are fixed through an open @@ -103,6 +116,7 @@ func (d Dirs) Ensure(uid uint32, o Owners) error { }{ {"outbox", int(uid), o.ListenerGID, 0o770 | os.ModeSetgid}, {"inbox", o.ListenerUID, o.ListenerGID, 0o700}, + {"check", o.ListenerUID, o.ListenerGID, 0o700}, } { if err := root.Mkdir(s.name, 0o700); err != nil && !errors.Is(err, fs.ErrExist) { return fmt.Errorf("make %s: %w", s.name, err) @@ -135,8 +149,9 @@ func seqName() string { } // writeIn writes data as a new file in dir, through os.Root: a random -// temporary name with O_EXCL|O_NOFOLLOW, then a rename in the same dir. -func writeIn(dir string, data []byte, mode os.FileMode) error { +// temporary name with O_EXCL|O_NOFOLLOW, then a rename in the same dir to +// final (a new sequence name when final is ""). +func writeIn(dir string, data []byte, mode os.FileMode, final string) error { root, err := os.OpenRoot(dir) if err != nil { return fmt.Errorf("open %s: %w", dir, err) @@ -157,8 +172,11 @@ func writeIn(dir string, data []byte, mode os.FileMode) error { if cerr := f.Close(); err == nil { err = cerr } + if final == "" { + final = seqName() + } if err == nil { - err = root.Rename(tmp, seqName()) + err = root.Rename(tmp, final) } if err != nil { _ = root.Remove(tmp) @@ -173,7 +191,7 @@ func (d Dirs) Send(m Message) error { if err != nil { return err } - return writeIn(d.Outbox(), b, 0o640) + return writeIn(d.Outbox(), b, 0o640, "") } // WriteApproval writes an approval to the inbox (the listener). @@ -182,7 +200,85 @@ func (d Dirs) WriteApproval(a Approval) error { if err != nil { return err } - return writeIn(d.Inbox(), b, 0o600) + return writeIn(d.Inbox(), b, 0o600, "") +} + +// WriteCheck writes a check request (the listener). It always has the same +// name, so many requests before root reads them are one file. +func (d Dirs) WriteCheck() error { + return writeIn(d.CheckDir(), []byte(CheckContent), 0o600, CheckFile) +} + +// HasCheck reports whether the check directory holds a file whose name +// does not start with ".". It reads and removes nothing. It opens the +// instance directory first and "check" inside it, so it never follows a +// symlink in place of "check". +func (d Dirs) HasCheck() bool { + inst, err := os.OpenRoot(d.inst()) + if err != nil { + return false + } + defer func() { _ = inst.Close() }() + root, err := inst.OpenRoot("check") + if err != nil { + return false + } + defer func() { _ = root.Close() }() + f, err := root.Open(".") + if err != nil { + return false + } + names, err := f.Readdirnames(-1) + _ = f.Close() + if err != nil { + return false + } + for _, n := range names { + if !strings.HasPrefix(n, ".") { + return true + } + } + return false +} + +// TakeChecks reads and removes every file in the check directory (root). +// It returns how many were valid check requests, and an error for each bad +// file. A valid request is a regular file named CheckFile, owned by the +// listener user, with exactly CheckContent in it. +func (d Dirs) TakeChecks(listenerUID int) (int, []error, error) { + root, err := os.OpenRoot(d.CheckDir()) + if err != nil { + if errors.Is(err, fs.ErrNotExist) { + return 0, nil, nil + } + return 0, nil, fmt.Errorf("open the check directory: %w", err) + } + defer func() { _ = root.Close() }() + names, err := list(root) + if err != nil { + return 0, nil, fmt.Errorf("read the check directory: %w", err) + } + n := 0 + var bad []error + for _, name := range names { + b, err := readOne(root, name, MaxCheck, listenerUID) + if rerr := root.RemoveAll(name); rerr != nil { + return n, bad, fmt.Errorf("remove %s/%s: %w. Fix: remove it as root", d.CheckDir(), name, rerr) + } + switch { + case err != nil: + case name != CheckFile: + err = errors.New("not the name of a check request") + case string(b) != CheckContent: + err = errors.New("not the content of a check request") + } + if err != nil { + bad = append(bad, fmt.Errorf("check file %s: %w", name, err)) + continue + } + n++ + } + return n, bad, nil } // list returns the names in dir, sorted, without names that start with ".". diff --git a/internal/spool/spool_test.go b/internal/spool/spool_test.go index 92493b0..fe5fb17 100644 --- a/internal/spool/spool_test.go +++ b/internal/spool/spool_test.go @@ -139,3 +139,87 @@ func TestOldTempFilesAreRemoved(t *testing.T) { t.Fatal("an old temp file must be removed") } } + +func TestCheckRequestRoundTrip(t *testing.T) { + d := dirs(t) + if d.HasCheck() { + t.Fatal("an empty check directory has no request") + } + // Many requests before root reads them are one file. + for range 3 { + if err := d.WriteCheck(); err != nil { + t.Fatal(err) + } + } + if !d.HasCheck() { + t.Fatal("the request must be seen") + } + n, bad, err := d.TakeChecks(os.Geteuid()) + if err != nil || n != 1 || len(bad) != 0 { + t.Fatalf("n=%d bad=%v err=%v", n, bad, err) + } + if n, _, _ := d.TakeChecks(os.Geteuid()); n != 0 { + t.Fatal("a request is used once") + } +} + +func TestBadCheckRequests(t *testing.T) { + putCheck := func(t *testing.T, d Dirs, name, body string) { + t.Helper() + if err := os.WriteFile(filepath.Join(d.CheckDir(), name), []byte(body), 0o600); err != nil { + t.Fatal(err) + } + } + cases := map[string]func(t *testing.T, d Dirs) int{ + "symlink": func(t *testing.T, d Dirs) int { + if err := os.Symlink("/etc/hostname", filepath.Join(d.CheckDir(), CheckFile)); err != nil { + t.Fatal(err) + } + return os.Geteuid() + }, + "wrong owner": func(t *testing.T, d Dirs) int { + putCheck(t, d, CheckFile, CheckContent) + return os.Geteuid() + 1 // the listener is another user + }, + "too big": func(t *testing.T, d Dirs) int { + putCheck(t, d, CheckFile, string(make([]byte, MaxCheck+1))) + return os.Geteuid() + }, + "other content": func(t *testing.T, d Dirs) int { + putCheck(t, d, CheckFile, "check config-repo\n") + return os.Geteuid() + }, + "other name": func(t *testing.T, d Dirs) int { + putCheck(t, d, "1.json", CheckContent) + return os.Geteuid() + }, + "directory": func(t *testing.T, d Dirs) int { + if err := os.MkdirAll(filepath.Join(d.CheckDir(), CheckFile, "x"), 0o700); err != nil { + t.Fatal(err) + } + return os.Geteuid() + }, + "fifo": func(t *testing.T, d Dirs) int { + if err := syscall.Mkfifo(filepath.Join(d.CheckDir(), CheckFile), 0o600); err != nil { + t.Skip(err) + } + return os.Geteuid() + }, + } + for n, c := range cases { + t.Run(n, func(t *testing.T) { + d := dirs(t) + uid := c(t, d) + got, bad, err := d.TakeChecks(uid) + if err != nil { + t.Fatal(err) + } + if got != 0 || len(bad) != 1 { + t.Fatalf("must be refused: n=%d bad=%v", got, bad) + } + if left, _ := os.ReadDir(d.CheckDir()); len(left) != 0 { + t.Fatal("a bad request must be removed") + } + }) + } +} diff --git a/internal/state/state.go b/internal/state/state.go index 839567e..4788650 100644 --- a/internal/state/state.go +++ b/internal/state/state.go @@ -70,6 +70,8 @@ type State struct { InProgress *InProgress `json:"in_progress,omitempty"` AskTimes []time.Time `json:"ask_times,omitempty"` LimitNote time.Time `json:"limit_note,omitempty"` + CheckAt time.Time `json:"check_at,omitzero"` // the last check request that was taken (DESIGN.md section 8) + CheckNote time.Time `json:"check_note,omitzero"` // the last "too many check requests" reply NextID int64 `json:"next_id"` } diff --git a/internal/telegram/faketg/faketg.go b/internal/telegram/faketg/faketg.go index c01009a..781c23a 100644 --- a/internal/telegram/faketg/faketg.go +++ b/internal/telegram/faketg/faketg.go @@ -60,11 +60,35 @@ func (s *Server) handle(w http.ResponseWriter, r *http.Request) { case "deleteWebhook": writeJSON(w, map[string]any{"ok": true, "result": true}) case "getUpdates": + // Like Telegram, give only the kinds in allowed_updates. + allowed := map[string]bool{} + if l, ok := req["allowed_updates"].([]any); ok { + for _, k := range l { + if ks, ok := k.(string); ok { + allowed[ks] = true + } + } + } + want := func(u telegram.Update) bool { + if u.UpdateID < num("offset") { + return false + } + switch { + case len(allowed) == 0: + return true + case u.CallbackQuery != nil: + return allowed["callback_query"] + case u.Message != nil: + return allowed["message"] + case u.EditedMessage != nil: + return allowed["edited_message"] + } + return false + } s.mu.Lock() - off := num("offset") var out []telegram.Update for _, u := range s.updates { - if u.UpdateID >= off { + if want(u) { out = append(out, u) } } @@ -76,7 +100,7 @@ func (s *Server) handle(w http.ResponseWriter, r *http.Request) { } s.mu.Lock() for _, u := range s.updates { - if u.UpdateID >= off { + if want(u) { out = append(out, u) } } @@ -159,6 +183,26 @@ func (s *Server) Tap(from, chat, msg int64, data string) { } } +// Say queues a message with text from user from in chat, as getUpdates +// gives it. With edited set, it comes as an edited message. +func (s *Server) Say(from, chat int64, text string, edited bool) { + s.mu.Lock() + s.nextID++ + m := &telegram.Message{MessageID: s.nextID, From: &telegram.User{ID: from}, Chat: telegram.Chat{ID: chat}, Text: text} + u := telegram.Update{UpdateID: s.nextUp, Message: m} + if edited { + m.EditDate = 3 + u = telegram.Update{UpdateID: s.nextUp, EditedMessage: m} + } + s.updates = append(s.updates, u) + s.nextUp++ + s.mu.Unlock() + select { + case s.wake <- struct{}{}: + default: + } +} + // EditByThief changes the text of a message as a token thief would. func (s *Server) EditByThief(msg int64, text string) { s.mu.Lock() diff --git a/internal/telegram/telegram.go b/internal/telegram/telegram.go index fe65435..64688c1 100644 --- a/internal/telegram/telegram.go +++ b/internal/telegram/telegram.go @@ -44,6 +44,7 @@ type Chat struct { // Message is a Telegram message. type Message struct { MessageID int64 `json:"message_id"` + From *User `json:"from,omitempty"` Chat Chat `json:"chat"` Text string `json:"text"` EditDate int64 `json:"edit_date,omitempty"` @@ -61,6 +62,8 @@ type CallbackQuery struct { type Update struct { UpdateID int64 `json:"update_id"` CallbackQuery *CallbackQuery `json:"callback_query,omitempty"` + Message *Message `json:"message,omitempty"` + EditedMessage *Message `json:"edited_message,omitempty"` } // Button is an inline keyboard button. @@ -152,11 +155,11 @@ func (c *Client) DeleteWebhook(ctx context.Context) error { return c.call(ctx, "deleteWebhook", map[string]any{"drop_pending_updates": false}, nil) } -// GetUpdates long-polls for button taps. +// GetUpdates long-polls for button taps and messages (for /check). func (c *Client) GetUpdates(ctx context.Context, offset int64, timeout int) ([]Update, error) { var ups []Update err := c.call(ctx, "getUpdates", map[string]any{ - "offset": offset, "timeout": timeout, "allowed_updates": []string{"callback_query"}, + "offset": offset, "timeout": timeout, "allowed_updates": []string{"callback_query", "message"}, }, &ups) return ups, err } diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index 66816fd..c517249 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -156,7 +156,32 @@ api_url = %q } return false }) + + // /check: the listener writes a request; the run (which the path unit + // starts on a host) takes it and says that there is no new commit. + // A /check from someone else writes nothing. + tg.Say(owner+1, chat, "/check", false) + tg.Say(owner, chat, "/check config-repo", false) + checkDir := filepath.Join(spool, "config-repo", "check") + wait(t, "the check request", func() bool { + es, _ := os.ReadDir(checkDir) + return len(es) == 1 + }) + if out := pg("run", "config-repo"); !strings.Contains(out, "result=nothing") { + t.Fatalf("run on /check: %s", out) + } + wait(t, "the check reply", func() bool { + checking, none := false, false + for _, m := range tg.Messages() { + checking = checking || strings.Contains(m.Text, "Checking config-repo") + none = none || strings.Contains(m.Text, "No new commit. The applied commit is "+second) + } + return checking && none + }) stop() + if strings.Contains(logs.String(), "/check") { + t.Fatal("the listener must not log the text of a message") + } if strings.Contains(logs.String(), token) { t.Fatal("the listener logged the token") }