diff --git a/CHANGELOG.md b/CHANGELOG.md index ca8c22d..158e566 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,39 @@ codes documented in `AGENT.md` are the public API. ### Added +- **Job actions `react`, `read` and `view`.** `react` puts an emoji (one, + a random pick from a list, or a weighted pick) on the message, replacing + the account's previous reaction; it reads the chat up to that message + first, so nobody sees a reaction on a message still shown unread, and an + album gets one reaction. `read` marks the chat read up to the triggering + message with the right RPC for the peer (forum topics included), coalesces + per chat and waits a reading time on top of its delay; `mentions` and + `reactions` clear those badges. `view` counts a channel view, or marks a + voice or round note listened in a DM or group, and never consumes + view-once media unless told to. +- **Knobs on every action, or on the job as a default:** `delay` (a random + delay from when the event arrived, which never blocks the bus), `percent`, + `presence` (`leave`, `blip`, `session`, with optional `quiet_hours`), + `on_takeover` (what to drop when you act in the chat yourself from another + device) and `dry_run`. +- **`reply` shows "typing..." first** (about 40 characters a second, 2-15 s), + on by default, `typing: false` to turn it off. Long form + `reply: {text, typing, delay, filters, processors}`. +- **Pacing per account and action kind**, configurable in a top-level + `pacing:` block of `jobs.yaml`, with cautious defaults (a reaction every + 4 s and at most 300 an hour; reads and views every 2 s; forwards and + replies every 1.5 s). FLOOD_WAIT reschedules and slows the queue, + transient errors retry with backoff, permanent ones are counted. +- **Pending actions survive restarts and crashes** in + `accounts//pending.json` and resume at their original due time. + React, view and reply expire 24 h after the event (configurable); read and + forward never do. A message replayed after a crash is not acted on twice. +- **Filters `sender_is_contact` and `chat_is_new`.** +- **`tlgr job queue`** lists pending actions with their due time, and + `tlgr job queue cancel` drops them by id, `--chat`, `--job` or `--all`. + `job list`, `job get` and `daemon status` report per-action counters: + done, skipped, superseded, expired, pending and errors. + - **`chat poster list` can walk deeper than one call.** A single call stops at `--max-messages 20000` and at the operation deadline, so a longer history could not be harvested at all: a bigger number was refused and @@ -29,6 +62,27 @@ codes documented in `AGENT.md` are the public API. messages were paced, so a chain of small calls would have read history unthrottled. +### Changed + +- **Every job action now runs through the op layer and is paced.** `forward` + and `reply` used to call Telethon directly from the bus handler; they now + run as `message.forward`, `message.send` and `media.upload` operations, so + a job obeys the policy allow/deny list, the rate limiter and the flood + budget like the CLI does. Their meaning is unchanged: a native forward + (with `drop_author`), or with processors a re-send of the processed text + or media caption; a reply to the matched message with processors applied. + Forwards on one account are now spaced at least 1.5 s apart, a + link-preview post is re-sent as text instead of failing, and an album gets + one reply instead of one per photo. +- **`jobs.yaml` is validated strictly.** An unknown key, a bad duration, a + percent outside 0-100 or an unknown presence mode is reported with the + job's name and the action's position, by `job add`, `job reload + --validate-only` and `config validate`. A broken job is skipped at load + while the others run, and a job whose edit broke it keeps running on + `job reload`. +- **`job get` runs in the daemon** so it can report live counters, and + `job disable` also drops what the job had queued. + ### Fixed - **Webhook filters work.** Both halves of `[webhook.filters]` were dead. diff --git a/docs/design/JOB_ACTIONS.md b/docs/design/JOB_ACTIONS.md new file mode 100644 index 0000000..134659f --- /dev/null +++ b/docs/design/JOB_ACTIONS.md @@ -0,0 +1,207 @@ +# Job actions: one execution model + +Status: implemented (feat/job-actions). Reference docs: `tlgr/gateway/README.md` +and `tlgr/actions/README.md`. This note records the choices the spec left +open, and why. + +## Shape + +A gateway job no longer runs actions in its bus handler. The handler +evaluates filters, rolls `percent`, calls the action's `plan()` (processors, +emoji pick, nothing awaited) and queues `PendingItem`s on the account's +`ActionScheduler` (`gateway/scheduler.py`). The scheduler owns delays, +pacing, quiet hours, expiry, takeover, retries and persistence, and calls the +action's `execute()`, which runs operations through `dispatch.execute` via +`DaemonOpRunner` (`gateway/executor.py`). One scheduler per account, shared by +every job on it; one worker and one `Pacer` per action kind. + +Every built-in action, `forward` and `reply` included, goes through this +path. A plain function registered with `@register_action` still runs at once, +outside the scheduler, so third-party actions written against the old +interface keep working. + +## Decisions + +### Pacing and jitter + +* `every` is a floor. The gap is drawn from `[every, 1.5 x every]`, a mean of + 1.25x and about +-20% around it. A window centred on `every` (the spec's + "about +-25%") would put reactions 3 s apart under a 4 s rule; a floor + keeps the promise "never closer than `every`". +* `per_hour` is a rolling hour of consumed slots. An absent `per_hour` in + `pacing:` keeps the default cap (300 for react), `per_hour: null` lifts it. +* Pacer state (the rolling hour, the slow-down) is not persisted; a restart + starts the hour afresh. The server-side flood memory (`flood.json`) still + survives, which is the part that matters. +* Due times are wall-clock, sleeps are monotonic, so a worker never sleeps + more than 60 s before looking at the clock again; a delay that spans a Mac + sleep fires at most a minute late. +* A request cancelled by Telethon (it cancels everything in flight when the + client disconnects) is a transient failure of that action, never the end of + the worker; a worker that ends anyway is restarted. +* Dry-run items skip the pacer and presence: they never touch Telegram, so + they should not slow the real actions on the same account. + +### Failures + +Classification goes through `core.errors.classify`, the same table the CLI +uses. + +* `RATE_LIMITED` (FLOOD_WAIT, slow mode): a job op sleeps a wait of up to + 10 s inside the request (`flood_wait_max=10`); a longer one comes back and + the item is rescheduled at `now + wait + 1..5 s`. The kind's pacer is held + past the wait and its spacing doubles (up to 8x), recovering after ten + minutes without another flood. An item gives up after 10 floods (an error). + Floods are counted apart from transient failures, so one does not use up + the other's retries. +* Retryable (`RETRYABLE`: network, timeouts, server errors, disconnected): + three retries after about 5 s, 30 s and 2 min (each x1-1.5). An action + that never expires (forward, read) then keeps retrying every ten minutes, + up to 12 attempts in all (about 1.5 h), so a relay rides out an account + that is reconnecting instead of dropping posts. +* Everything else is permanent and counted under `errors` with `last_error` + (`USAGE: REACTION_INVALID: ...`): REACTION_INVALID, MESSAGE_ID_INVALID, + CHAT_WRITE_FORBIDDEN, policy denials, PEER_FLOOD and frozen accounts. The + last two are deliberately not retried: the account has been told to stop. + +### Persistence + +* A JSON file per account, `accounts//pending.json`, written with + `write_private` (temp file, chmod 0600, fsync, rename). Chosen over SQLite + because the queue is small, it matches `flood.json` beside it, and it can be + read with `jq` when diagnosing. Writes are debounced (1 s) and forced at + shutdown. +* Bounded at 10 000 items per account; past that a new item is refused and + counted as an error, logged once. +* Items carry everything needed to run without the Telethon event: chat, + message, peer kind, forum topic, `grouped_id`, presence, quiet hours, + takeover mode, dry run, and the payload (emoji, processed text, forward + destination, view mode). +* The file also keeps the keys of the last 5 000 finished items. A key is + `job|index|event|chat|msg|extra` (the edit time for an edit event, the + destination for a forward), so an update replayed by catch-up after a crash + (the update state is saved once a minute) is not acted on twice. This also + protects forwards, which could duplicate before. +* Resume happens once per scheduler, when its account's jobs are created at + boot. Items whose job no longer exists (or is disabled) are dropped and + logged. An item that was running when the daemon died runs again + (at-least-once); a clean shutdown stops starting new actions, lets a + running one finish within the drain, then saves. +* `job remove` drops the job's pending items and counters; `job disable` + drops its pending items (counted as superseded). A job updated by + `job reload` keeps its queue. + +### Expiry + +React, view and reply expire 24 h after the event arrived; read and forward +never. `pacing..expire` overrides per kind with a duration, or +`never`/`null`/`0`. + +### react + +* Custom emoji are supported as `custom:`, the spelling + `reaction add` already uses. +* Album target: the message carrying the caption, else the first one. That is + where Telegram Desktop attaches an album's reactions + (`HistoryView::GroupedMedia::itemForText()`: the caption item, else the + first part) and what Telegram for Android uses + (`MessageObject.GroupedMessages.findPrimaryMessageObject()`). If the + caption message arrives after the first one while the reaction is still + pending, the pending reaction is moved to it. +* An album is one `percent` roll, not one per photo. Album siblings count as + `skipped`. +* React implies read through `ensure_read`: paced on the read queue, read up + to the reacted message, and any pending read for that chat at or below it + finishes as done (coalesced). If the chat is already known read that far + (by tlgr or by you elsewhere) no read is sent. If the read fails the + reaction does not go out. + +### reply + +* One reply per album as well (to the caption message), the same rule as + react. Replying once per photo was never useful in a DM, which is the + main use case. +* Typing is a two-step item: `chat.typing` is started in the background and + the item is re-queued for when typing ends, then `message.send` goes out. + The reply queue is never blocked for the typing time. +* Text is sent as markdown, which is what `event.reply()` did with the + client's default parse mode. + +### forward + +* One pending item per destination, so one refusing destination does not + stop the others and each is retried or counted on its own. +* With processors: a photo or document is re-sent with `media.upload + --from-message` and the processed caption; a link-preview post is re-sent + as text (Telegram regenerates the preview), where the old code failed on + it; other media with text is re-sent as text, without text it is skipped. + Text is taken as `message.text` (markdown under the client's parse mode) + and sent with `--parse md`, so formatting survives as it did. +* Albums are still forwarded message by message, as before; grouping them + would need a hold window and changes what the destination sees. + +### read + +Reading time is 250 words a minute plus 3 s for media, capped at 60 s, added +to the `delay` draw. + +### view + +A broadcast channel post (the `post` flag, or a broadcast chat entity) is a +view. In private chats and groups only voice and round notes are something +to view; view-once media needs `include_view_once: true`. + +### Presence + +* The rule for two jobs with different modes on one account: the most online + request wins while its actions run. `leave` never sends anything and never + ends an online stretch another job started; offline is sent only once no + `blip`/`session` action is running and, for `session`, none is due within + 5 s. Linger after the last action: 5 s. +* `session` goes online when its first action executes, not ahead of it. +* If the account-level `[presence] mode` is not `off`, the daemon already + manages the status and job presence is silent. +* Quiet hours use `[defaults] timezone` (an IANA name) when set, else the + daemon's local time. Held items are released at the end of the window, + spread over five minutes, then go through the pacer. + +### Manual takeover + +* A read is tlgr's own when its `max_id` is at or below the highest id tlgr + asked to read in that chat; the mark is set before the RPC, because the + echo can arrive before the answer. +* An outgoing message is tlgr's own when its id is one tlgr recorded sending, + or when tlgr sent in that chat less than 10 s earlier. +* A read that arrives within 10 s of tlgr sending in that chat is also tlgr's + own: sending marks the chat read on the server. +* Takeover drops items in that chat with a message id up to the read or sent + id, by each item's own `on_takeover`; forwards are never dropped. A read in + a forum topic only drops items of that topic (forum ids span the chat). +* `job queue cancel` counts what it drops as `superseded` too. + +### Filters + +* `sender_is_contact` trusts the `contact` flag on the sender entity the + update carried; without one it fetches the sender and caches the answer for + 10 minutes. +* `chat_is_new` probes the history once per chat (`get_messages(limit=1, + offset_id=msg)`) and caches the first message id (or "not new"), in an LRU + of 5 000 chats per process. +* Both are coroutines, so filter evaluation gained `evaluate_async`; the + synchronous `evaluate` (webhook filters) rejects them with a reason. + +### Validation and the CLI + +* Validation is strict per job: unknown keys, bad durations, percent outside + 0-100, unknown presence or takeover modes, unknown actions, processors on + an action that cannot use them. At load a broken job is skipped and logged + while the others run; on `job reload` a job whose edit broke it keeps + running in its last good form. +* `tlgr job queue` lists and `tlgr job queue cancel` cancels. The registry + forbids an op id that is also a group, so the list op is `job.queue.list` + tagged `group-default`, and its CLI group runs it when called bare. Options + go on the leaf (`tlgr job queue list --job dm-ack`). +* `job.queue.cancel` is destructive as a whole, so every variant needs + `--yes` off a terminal, like `job remove`. +* `job.get` moved from the local surface to the daemon so it can report live + counters. diff --git a/docs/reference/PARITY.md b/docs/reference/PARITY.md index 306bf1e..27486e4 100644 --- a/docs/reference/PARITY.md +++ b/docs/reference/PARITY.md @@ -7,7 +7,7 @@ Coverage against the Telegram feature catalog, computed from the registry: every `covered` is implemented today. `acct%` is covered **plus** waived — an id this build genuinely cannot cover, named in `tlgr/data/parity_waivers.toml` with the reason and the MTProto method that is missing. Ids whose feasibility is `not-applicable` or `prohibited` are excluded from the denominator once and never counted again. ``` -catalog 2026-09-02 — 678 operations, 951 invocable paths +catalog 2026-09-02 — 680 operations, 953 invocable paths domain covered req % acct% ops auth_sessions_security 89 89 100.0% 100.0% 45 @@ -21,7 +21,7 @@ messages_core 167 167 100.0% 100.0% 61 polls_reactions_content 173 174 99.4% 100.0% 99 profile_settings_privacy 178 178 100.0% 100.0% 109 stories 120 120 100.0% 100.0% 48 -updates_sync_network 189 189 100.0% 100.0% 68 +updates_sync_network 189 189 100.0% 100.0% 70 priority covered req % acct% P0 178 178 100.0% 100.0% @@ -49,7 +49,7 @@ uncovered: 9 (9 waived with a reason) | `polls_reactions_content` | 173 | 174 | 99.4% | 100.0% | 99 | | `profile_settings_privacy` | 178 | 178 | 100.0% | 100.0% | 109 | | `stories` | 120 | 120 | 100.0% | 100.0% | 48 | -| `updates_sync_network` | 189 | 189 | 100.0% | 100.0% | 68 | +| `updates_sync_network` | 189 | 189 | 100.0% | 100.0% | 70 | ## By priority diff --git a/docs/reference/README.md b/docs/reference/README.md index 2c9a880..290edd5 100644 --- a/docs/reference/README.md +++ b/docs/reference/README.md @@ -2,7 +2,7 @@ # Command reference -678 operations across 47 groups, generated from the operation registry. Groups still served by v1's hand-written commands are not listed here; they arrive with their own PR. +680 operations across 47 groups, generated from the operation registry. Groups still served by v1's hand-written commands are not listed here; they arrive with their own PR. | Group | Operations | Reference | |---|---:|---| @@ -27,7 +27,7 @@ | `gift` | 19 | [gift.md](gift.md) | | `giveaway` | 6 | [giveaway.md](giveaway.md) | | `inline` | 7 | [inline.md](inline.md) | -| `job` | 8 | [job.md](job.md) | +| `job` | 10 | [job.md](job.md) | | `location` | 9 | [location.md](location.md) | | `media` | 28 | [media.md](media.md) | | `message` | 39 | [message.md](message.md) | diff --git a/docs/reference/job.md b/docs/reference/job.md index a76858a..d0f3a87 100644 --- a/docs/reference/job.md +++ b/docs/reference/job.md @@ -2,15 +2,17 @@ # `tlgr job` -8 operations. Every one takes the global flags (`--json`, `--plain`, `-a/--account`, `--results-only`, `--select`, `--dry-run`, `--yes`, `--no-input`, `--flood-wait-max`, `-v`) anywhere on the line. +10 operations. Every one takes the global flags (`--json`, `--plain`, `-a/--account`, `--results-only`, `--select`, `--dry-run`, `--yes`, `--no-input`, `--flood-wait-max`, `-v`) anywhere on the line. | Command | Summary | |---|---| | [`job add`](#tlgr-job-add) | Add a gateway job | | [`job disable`](#tlgr-job-disable) | Disable a job without removing it | | [`job enable`](#tlgr-job-enable) | Enable a disabled job | -| [`job get`](#tlgr-job-get) | Show one job's resolved pipeline (filters, processors, actions) | +| [`job get`](#tlgr-job-get) | Show one job's resolved pipeline (filters, processors, actions) and counters | | [`job list`](#tlgr-job-list) | List gateway jobs and their state | +| [`job queue cancel`](#tlgr-job-queue-cancel) | Cancel pending job actions by id, chat, job, or all | +| [`job queue list`](#tlgr-job-queue-list) | List pending job actions (delayed, paced or held by quiet hours) | | [`job reload`](#tlgr-job-reload) | Hot-reload jobs.yaml without restarting the daemon | | [`job remove`](#tlgr-job-remove) | Remove a job | | [`job test`](#tlgr-job-test) | Dry-run a job's filters against real or synthetic events | @@ -36,6 +38,7 @@ tlgr job add [OPTIONS] | `--filter` | text | | Filter entry (repeatable). | | `--from-file` | text | | Read one job (or a jobs list) from YAML/JSON. | | `--for-account` | text | | Account the job runs on. | +| `--knob` | text | | Job-level action default: delay, percent, presence, on_takeover, dry_run. | | `--name` | text | | Job name. | | `--processor` | text | | Processor entry (repeatable). | @@ -105,7 +108,7 @@ toggles a rule; the filtering itself is the gateway's. ### `job get` -Show one job's resolved pipeline (filters, processors, actions). +Show one job's resolved pipeline (filters, processors, actions) and counters. ``` tlgr job get [OPTIONS] @@ -163,6 +166,72 @@ lists the rules; proving one fires is `job test`. +### `job queue cancel` + +Cancel pending job actions by id, chat, job, or all. + +Selectors combine: `--job dm-ack --chat @alice` drops only that job's actions in that chat. A cancelled action counts as `superseded`. + +``` +tlgr job queue cancel [ID]... [OPTIONS] +``` + +**mutating · destructive (needs `--yes` off a TTY) · runs without an account · returns `QueueCancel`** + +| Argument | Type | Required | Meaning | +|---|---|---|---| +| `ID` | text | any number | Pending action id(s). | + +| Flag | Type | Default | Meaning | +|---|---|---|---| +| `--chat` | text | | Everything pending in this chat. | +| `--all` | flag | | Every pending action on every account. | +| `--job` | text | | Everything this job has. | + +```console +$ tlgr job queue cancel --job dm-ack --yes --json +``` + +
Catalog coverage (0 full, 1 partial) + +Partial: `updates.stream-event-filtering` + +cancels what the rules scheduled; the filtering itself is the gateway's. + +
+ +### `job queue list` + +List pending job actions (delayed, paced or held by quiet hours). + +Every action a job has scheduled and not yet run, across accounts, soonest first. `state` is `waiting` before the due time, `due` while it waits for its pacer slot, and `running` while it talks to Telegram. + +``` +tlgr job queue list [OPTIONS] +``` + +**paginated (`LOCAL` cursor) · idempotent (reports `already`) · runs without an account · returns `Page[PendingAction]`** + +| Flag | Type | Default | Meaning | +|---|---|---|---| +| `--action` | text | | Only this action (react, read, ...). | +| `--chat` | text | | Only this chat (@name or marked id). | +| `--job` | text | | Only this job. | + +Pagination is transport-level: `--limit/-n`, `--cursor TOKEN`, `--all` (walked inside the daemon, paced by the account's own rate limiter). + +```console +$ tlgr job queue list --job dm-ack --json +``` + +
Catalog coverage (0 full, 1 partial) + +Partial: `updates.stream-event-filtering` + +shows what the rules scheduled; the filtering itself is the gateway's. + +
+ ### `job reload` Hot-reload jobs.yaml without restarting the daemon. diff --git a/jobs.example.yaml b/jobs.example.yaml index b6b4388..1a6a6b9 100644 --- a/jobs.example.yaml +++ b/jobs.example.yaml @@ -4,8 +4,50 @@ # event -> filters -> processors -> actions # # See tlgr/gateway/README.md for the full reference. +# +# Every action runs through tlgr's op layer and is paced per account and +# action kind. The optional `pacing:` block changes the pace; the values +# below are the defaults. + +pacing: + main: + react: {every: 4s, per_hour: 300} + read: {every: 2s} + view: {every: 2s} + forward: {every: 1.5s} + reply: {every: 1.5s} + expire: {react: 24h, view: 24h, reply: 24h} jobs: + # Acknowledge DMs from contacts the way a person would: read after a + # while, listen to voice notes, sometimes react, and answer a new chat. + - name: dm-ack + account: main + filters: {chat_type: private, sender_is_contact: true} + presence: {mode: session, quiet_hours: "01:00-08:00"} + on_takeover: cancel # cancel | cancel_read | ignore + actions: + - read: {delay: 10-90s} # + reading time + - view: {delay: 15-120s} # voice/round notes + - react: + emoji: ["👍", "❤", "🔥"] # random; or {"👍": 3, "🔥": 1} + percent: 60 + delay: 30-300s + - reply: + text: "Got it, will answer soon" + filters: {chat_is_new: true} + typing: true # default for reply + delay: 1-3m + + # Try a rule without touching Telegram: it is scheduled, logged and + # counted, never sent. + - name: dry-run-example + account: main + dry_run: true + filters: {chat_type: private} + actions: + - react: "👀" + # Auto-reply to all private messages - name: private-bot account: main diff --git a/tests/fake_clock.py b/tests/fake_clock.py new file mode 100644 index 0000000..558278e --- /dev/null +++ b/tests/fake_clock.py @@ -0,0 +1,58 @@ +"""A virtual clock for the job-action scheduler: time moves only when a test says. + +`sleep()` parks the caller on a heap of wake times; `advance()` walks the heap +in order, setting `now` to each wake time before releasing the sleeper, and +lets the event loop run between steps. A pacer that waits four seconds +therefore waits exactly four virtual seconds, and a test about a whole hour of +reactions takes milliseconds. +""" + +from __future__ import annotations + +import asyncio +import heapq +import itertools + +__all__ = ["FakeClock", "settle"] + + +async def settle(rounds: int = 200) -> None: + """Let every runnable task run until the loop goes quiet.""" + for _ in range(rounds): + await asyncio.sleep(0) + + +class FakeClock: + def __init__(self, start: float = 1_800_000_000.0) -> None: + self._now = float(start) + self._sleepers: list[tuple[float, int, asyncio.Future[None]]] = [] + self._seq = itertools.count() + + def now(self) -> float: + return self._now + + async def sleep(self, seconds: float) -> None: + future: asyncio.Future[None] = asyncio.get_running_loop().create_future() + heapq.heappush(self._sleepers, (self._now + max(0.0, seconds), next(self._seq), future)) + await future + + async def advance(self, seconds: float, *, rounds: int = 200) -> None: + target = self._now + seconds + await settle(rounds) + while self._sleepers and self._sleepers[0][0] <= target: + wake, _, future = heapq.heappop(self._sleepers) + if future.done(): + continue + self._now = max(self._now, wake) + future.set_result(None) + await settle(rounds) + self._now = target + await settle(rounds) + + async def run_for(self, seconds: float, *, step: float = 0.5) -> None: + """Advance in small steps, so work that sleeps again keeps getting woken.""" + elapsed = 0.0 + while elapsed < seconds: + chunk = min(step, seconds - elapsed) + await self.advance(chunk) + elapsed += chunk diff --git a/tests/fake_telethon.py b/tests/fake_telethon.py index ec06535..8e8d8c6 100644 --- a/tests/fake_telethon.py +++ b/tests/fake_telethon.py @@ -1722,8 +1722,13 @@ def _raw_DeleteParticipantHistoryRequest(self, request: Any) -> Any: return self._affected_history() def _raw_ReadHistoryRequest(self, request: Any) -> Any: - chat_id = self._chat_id(request.peer) + # `messages.readHistory` and `channels.readHistory` share a class + # name; the channel one carries `channel` and answers a bare bool. + channel = getattr(request, "channel", None) + chat_id = self._chat_id(channel if channel is not None else request.peer) self.world.read_inbox[chat_id] = int(request.max_id) + if channel is not None: + return True return types.messages.AffectedMessages(pts=1, pts_count=0) def _raw_SaveDraftRequest(self, request: Any) -> bool: diff --git a/tests/job_helpers.py b/tests/job_helpers.py new file mode 100644 index 0000000..08e5427 --- /dev/null +++ b/tests/job_helpers.py @@ -0,0 +1,220 @@ +"""Shared pieces for the job-action tests: real TL updates, a recording runner. + +The updates are real `telethon.tl.types` objects, built the way the server +sends them, and turned into the high-level event a job sees with the same +`build_event` the daemon uses. The `RecordingRunner` stands in for the op +layer where a test is about scheduling rather than about what an operation +sends; the integration tests run the real operations against the fake client. +""" + +from __future__ import annotations + +import random +from types import SimpleNamespace +from typing import Any + +from fake_clock import FakeClock +from telethon.tl import types + +from tlgr.gateway.engine import Gateway, _builders +from tlgr.gateway.event import Event +from tlgr.gateway.scheduler import ActionScheduler +from tlgr.gateway.tlevents import build_event + +ALICE = 4242 +BOB = 4343 +CHANNEL = 5150 +CHANNEL_ID = -1000000000000 - CHANNEL +GROUP = 777 +GROUP_ID = -GROUP + + +class RecordingRunner: + """An op runner that records `(op, request, now)` and can be told to fail.""" + + def __init__(self, clock: Any = None) -> None: + self.calls: list[tuple[str, dict[str, Any], float]] = [] + self.clock = clock + self.failures: dict[str, list[BaseException]] = {} + self.results: dict[str, Any] = {} + + def fail(self, op: str, *errors: BaseException) -> None: + self.failures.setdefault(op, []).extend(errors) + + async def __call__(self, op: str, request: dict[str, Any]) -> Any: + now = self.clock.now() if self.clock is not None else 0.0 + self.calls.append((op, dict(request), now)) + pending = self.failures.get(op) + if pending: + raise pending.pop(0) + result = self.results.get(op) + if callable(result): + return result(request) + return result + + def ops(self, *names: str) -> list[str]: + return [op for op, _, _ in self.calls if not names or op in names] + + def requests(self, op: str) -> list[dict[str, Any]]: + return [request for name, request, _ in self.calls if name == op] + + def times(self, op: str) -> list[float]: + return [when for name, _, when in self.calls if name == op] + + +def make_scheduler( + *, clock: FakeClock | None = None, seed: int = 1, store: Any = None, **kwargs: Any +) -> tuple[ActionScheduler, RecordingRunner, FakeClock]: + clock = clock or FakeClock() + runner = RecordingRunner(clock) + scheduler = ActionScheduler( + "work", runner, clock=clock, rng=random.Random(seed), store=store, **kwargs + ) + return scheduler, runner, clock + + +def private_update( + message_id: int, + text: str = "hello", + *, + user: int = ALICE, + out: bool = False, + media: Any = None, + grouped_id: int | None = None, +) -> types.UpdateNewMessage: + message = types.Message( + id=message_id, + peer_id=types.PeerUser(user_id=user), + from_id=None if out else types.PeerUser(user_id=user), + date=None, + message=text, + out=out, + media=media, + grouped_id=grouped_id, + ) + return types.UpdateNewMessage(message=message, pts=1, pts_count=1) + + +def channel_update( + message_id: int, text: str = "post", *, channel: int = CHANNEL, media: Any = None +) -> types.UpdateNewChannelMessage: + message = types.Message( + id=message_id, + peer_id=types.PeerChannel(channel_id=channel), + date=None, + message=text, + post=True, + media=media, + ) + return types.UpdateNewChannelMessage(message=message, pts=1, pts_count=1) + + +def group_update(message_id: int, text: str = "hi all", *, user: int = ALICE) -> Any: + message = types.Message( + id=message_id, + peer_id=types.PeerChat(chat_id=GROUP), + from_id=types.PeerUser(user_id=user), + date=None, + message=text, + ) + return types.UpdateNewMessage(message=message, pts=1, pts_count=1) + + +def topic_update(message_id: int, topic: int, *, channel: int = CHANNEL) -> Any: + message = types.Message( + id=message_id, + peer_id=types.PeerChannel(channel_id=channel), + from_id=types.PeerUser(user_id=ALICE), + date=None, + message="in a topic", + reply_to=types.MessageReplyHeader( + reply_to_msg_id=topic, forum_topic=True, reply_to_top_id=None + ), + ) + return types.UpdateNewChannelMessage(message=message, pts=1, pts_count=1) + + +def voice_media(*, ttl: int | None = None, round_video: bool = False) -> Any: + attributes: list[Any] + if round_video: + attributes = [types.DocumentAttributeVideo(duration=3, w=240, h=240, round_message=True)] + mime = "video/mp4" + else: + attributes = [types.DocumentAttributeAudio(duration=3, voice=True)] + mime = "audio/ogg" + document = types.Document( + id=99, + access_hash=1, + file_reference=b"", + date=None, + mime_type=mime, + size=10, + dc_id=2, + attributes=attributes, + ) + return types.MessageMediaDocument(document=document, ttl_seconds=ttl) + + +def photo_media() -> Any: + photo = types.Photo(id=77, access_hash=1, file_reference=b"", date=None, sizes=[], dc_id=2) + return types.MessageMediaPhoto(photo=photo) + + +async def tg_event(update: Any, client: Any = None) -> Any: + """The high-level Telethon event a job's filters and actions read.""" + if client is None: + from fake_telethon import FakeTelegramClient, World + + client = FakeTelegramClient(World()) + built = await build_event(_builders(["new_message"]), update, client) + assert built is not None, "the builder declined the update" + return built[0] + + +async def deliver(job: Gateway, update: Any, client: Any = None) -> None: + """Hand *update* to a job the way its bus handler does.""" + await job._handle(await tg_event(update, client), "new_message") + + +def envelope(event: Any, account: str = "work") -> Event: + return Event(source="telegram", raw=event, account=account) + + +def bus_envelope(account: str = "work", event_type: str = "message_new") -> Any: + return SimpleNamespace(type=event_type, account=account) + + +async def daemon_job_env(daemon: Any, alias: str = "work", *, seed: int = 1) -> Any: + """Connect *alias* and give it a scheduler on a virtual clock. + + The scheduler runs the real operations through the daemon's dispatcher + against the fake client, so a test asserts on the requests that reached + "Telegram" (`world.calls`), not on a mock. + """ + from tlgr.gateway.executor import DaemonOpRunner + + await daemon.sessions.connect_all([alias]) + clock = FakeClock() + scheduler = ActionScheduler( + alias, DaemonOpRunner(daemon, alias), clock=clock, rng=random.Random(seed) + ) + daemon._schedulers[alias] = scheduler + daemon.bus.add_handler(scheduler.on_bus) + scheduler.start() + return SimpleNamespace( + scheduler=scheduler, clock=clock, client=daemon.get_client(alias), alias=alias + ) + + +async def until(predicate: Any, *, clock: FakeClock | None = None, timeout: float = 5.0) -> None: + """Wait (really, briefly) for work running through the dispatcher to land.""" + import asyncio + + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout + while not predicate(): + if loop.time() > deadline: + raise AssertionError("condition never became true") + if clock is not None: + await clock.advance(0.5) + await asyncio.sleep(0.01) diff --git a/tests/test_actions.py b/tests/test_actions.py index a106ce7..dc463b7 100644 --- a/tests/test_actions.py +++ b/tests/test_actions.py @@ -1,24 +1,26 @@ -"""Tests for the action registry.""" +"""The action registry, and each built-in action's config and planning.""" from __future__ import annotations -from unittest.mock import AsyncMock, MagicMock +import random import pytest -from tlgr.actions import get_action, list_actions, register_action -from tlgr.gateway.event import Event +from tlgr.actions import get_action, get_builtin, list_actions, register_action +from tlgr.actions.base import Action, ActionError, MessageFacts from tlgr.processors import ProcessorChain class TestRegistry: def test_builtin_actions_registered(self): names = list_actions() - assert "reply" in names - assert "forward" in names + for name in ("reply", "forward", "react", "read", "view"): + assert name in names + assert isinstance(get_action(name), Action) def test_get_unknown_returns_none(self): assert get_action("nonexistent") is None + assert get_builtin("nonexistent") is None def test_custom_action_registration(self): @register_action("_test_noop") @@ -26,87 +28,119 @@ async def noop(event, config, client, chain=None): pass assert get_action("_test_noop") is not None + assert get_builtin("_test_noop") is None -def _make_event(text="hello"): - msg = MagicMock() - msg.text = text - msg.message = text - msg.sender_id = 100 - msg.media = None - msg.action = None - - tg_event = MagicMock() - tg_event.message = msg - tg_event.reply = AsyncMock() - tg_event.chat_id = 42 - - return Event(source="telegram", raw=tg_event, account="test") - - -def _make_client(): - client = MagicMock() - client.resolve_chat = AsyncMock(return_value=999) - client.client = MagicMock() - client.client.forward_messages = AsyncMock() - client.client.send_message = AsyncMock() - client.client.send_file = AsyncMock() - return client - - -class TestReplyAction: - @pytest.mark.asyncio - async def test_reply_string(self): - action = get_action("reply") - ev = _make_event() - client = _make_client() - await action(ev, "hello!", client, None) - ev.raw.reply.assert_awaited_once_with("hello!") - - @pytest.mark.asyncio - async def test_reply_with_processors(self): - action = get_action("reply") - ev = _make_event() - client = _make_client() - chain = ProcessorChain().add("add_prefix", {"prefix": "[BOT]"}) - await action(ev, "hello!", client, chain) - ev.raw.reply.assert_awaited_once() - call_text = ev.raw.reply.call_args[0][0] - assert "[BOT]" in call_text - - @pytest.mark.asyncio - async def test_reply_non_telegram(self): - action = get_action("reply") - ev = Event(source="webhook", raw={}, account="test") - client = _make_client() - await action(ev, "hello!", client, None) - # Should not crash, just log a warning - - -class TestForwardAction: - @pytest.mark.asyncio - async def test_forward_simple(self): - action = get_action("forward") - ev = _make_event() - client = _make_client() - await action(ev, {"to": "@dest"}, client, None) - client.client.forward_messages.assert_awaited_once() - - @pytest.mark.asyncio - async def test_forward_with_processors(self): - action = get_action("forward") - ev = _make_event(text="hello world") - client = _make_client() +def _facts(**kwargs): + return MessageFacts(chat_id=-1001234, msg_id=5, **{"text": "hello", **kwargs}) + + +RNG = random.Random(0) + + +class TestForward: + action = get_builtin("forward") + + def test_short_and_long_forms(self): + assert self.action.parse("@dest") == {"to": ["@dest"], "drop_author": False} + assert self.action.parse({"to": ["@a", -1001], "drop_author": True}) == { + "to": ["@a", "-1001"], + "drop_author": True, + } + + @pytest.mark.parametrize( + "config", [{}, {"to": []}, {"to": "@a", "drop_author": "yes"}, {"to": "@a", "x": 1}] + ) + def test_invalid(self, config): + with pytest.raises(ActionError): + self.action.parse(config) + + def test_a_native_forward_per_destination(self): + params = self.action.parse({"to": ["@a", "@b"], "drop_author": True}) + payloads = self.action.plan(_facts(), params, None, RNG) + assert payloads == [ + {"to": "@a", "drop_author": True, "resend": None}, + {"to": "@b", "drop_author": True, "resend": None}, + ] + + def test_processors_turn_it_into_a_re_send(self): chain = ProcessorChain().add("add_prefix", {"prefix": "[FWD]"}) - await action(ev, {"to": ["@dest"]}, client, chain) - client.client.send_message.assert_awaited_once() - call_text = client.client.send_message.call_args[0][1] - assert "[FWD]" in call_text - - @pytest.mark.asyncio - async def test_forward_string_config(self): - action = get_action("forward") - ev = _make_event() - client = _make_client() - await action(ev, "@dest", client, None) - client.client.forward_messages.assert_awaited_once() + params = self.action.parse({"to": "@a"}) + text = self.action.plan(_facts(), params, chain, RNG)[0]["resend"] + assert text == {"text": "[FWD]\nhello", "media": False} + photo = self.action.plan(_facts(media="photo"), params, chain, RNG)[0]["resend"] + assert photo["media"] is True + preview = self.action.plan(_facts(media="webpage"), params, chain, RNG)[0]["resend"] + assert preview["media"] is False + + @pytest.mark.parametrize( + "facts", + [ + {"service": True}, + {"view_once": True, "media": "photo"}, + {"text": "", "media": None}, + ], + ) + def test_what_was_never_forwardable_is_skipped(self, facts): + assert self.action.plan(_facts(**facts), self.action.parse("@a"), None, RNG) == [] + + +class TestReply: + action = get_builtin("reply") + + def test_short_and_long_forms(self): + assert self.action.parse("hi") == {"text": "hi", "typing": True} + assert self.action.parse({"text": "hi", "typing": False}) == { + "text": "hi", + "typing": False, + } + + @pytest.mark.parametrize("config", ["", {"typing": True}, {"text": "x", "typing": 1}, 5]) + def test_invalid(self, config): + with pytest.raises(ActionError): + self.action.parse(config) + + +class TestReact: + action = get_builtin("react") + + def test_forms(self): + assert self.action.parse("👍")["choices"] == [["👍", 1.0]] + assert self.action.parse(["👍", "❤"])["choices"] == [["👍", 1.0], ["❤", 1.0]] + assert self.action.parse({"👍": 3, "🔥": 1})["choices"] == [["👍", 3.0], ["🔥", 1.0]] + long = self.action.parse({"emoji": {"👍": 2}, "big": True}) + assert long == {"choices": [["👍", 2.0]], "big": True} + assert self.action.parse("custom:5368324170671202286")["choices"][0][0].startswith( + "custom:" + ) + + @pytest.mark.parametrize( + "config", + [[], {}, {"👍": 0}, {"👍": "lots"}, {"emoji": "👍", "size": 3}, "custom:abc", 7], + ) + def test_invalid(self, config): + with pytest.raises(ActionError): + self.action.parse(config) + + +class TestReadAndView: + def test_read_forms(self): + read = get_builtin("read") + assert read.parse(None) == read.parse(True) == read.parse({}) + assert read.parse({"mentions": True})["mentions"] is True + with pytest.raises(ActionError): + read.parse(False) + with pytest.raises(ActionError): + read.parse({"history": True}) + + def test_view_plans_by_peer_and_media(self): + view = get_builtin("view") + params = view.parse({}) + assert view.plan(_facts(peer="channel"), params, None, RNG) == [{"mode": "views"}] + voice = _facts(peer="user", voice_or_round=True) + assert view.plan(voice, params, None, RNG) == [{"mode": "contents"}] + assert view.plan(_facts(peer="user"), params, None, RNG) == [] + once = _facts(peer="user", voice_or_round=True, view_once=True) + assert view.plan(once, params, None, RNG) == [] + brave = view.parse({"include_view_once": True}) + assert view.plan(once, brave, None, RNG) == [{"mode": "contents"}] diff --git a/tests/test_daemon_jobs.py b/tests/test_daemon_jobs.py index 073d4a4..caee53b 100644 --- a/tests/test_daemon_jobs.py +++ b/tests/test_daemon_jobs.py @@ -68,3 +68,86 @@ async def test_a_broken_jobs_file_does_not_stop_the_daemon( finally: daemon.request_shutdown() await asyncio.wait_for(runner, timeout=10) + + +PACED = """\ +pacing: + {account}: + read: {{every: 3s}} +jobs: + - name: dm-read + account: {account} + filters: {{chat_type: private}} + actions: + - read: {{delay: 1h}} +""" + + +class TestPendingActionsAcrossRestarts: + async def test_a_pending_action_is_saved_at_shutdown_and_resumed_at_boot( + self, tlgr_home: Path, stub_account: str, world + ): + import json + + from fake_telethon import make_user + from job_helpers import ALICE, private_update + + from tlgr.daemon.app import Daemon + + world.add_user(make_user(ALICE, username="alice")) + (tlgr_home / "jobs.yaml").write_text(PACED.format(account=stub_account)) + + daemon = Daemon(tlgr_home, client_factory=fake_client_factory(world)) + runner = asyncio.create_task(daemon.run()) + try: + await _until(lambda: any(job.get("running") for job in daemon.list_jobs())) + scheduler = daemon.schedulers[stub_account] + assert scheduler.pacers["read"].rule.every == 3.0 + session = daemon.sessions.get(stub_account) + await session.client.feed(private_update(10)) + await _until(lambda: len(scheduler.items) == 1) + due = next(iter(scheduler.items.values())).due_at + finally: + daemon.request_shutdown() + await asyncio.wait_for(runner, timeout=10) + + saved = tlgr_home / "accounts" / stub_account / "pending.json" + assert saved.stat().st_mode & 0o777 == 0o600 + body = json.loads(saved.read_text()) + assert [item["msg_id"] for item in body["items"]] == [10] + + again = Daemon(tlgr_home, client_factory=fake_client_factory(world)) + runner = asyncio.create_task(again.run()) + try: + await _until(lambda: stub_account in again.schedulers) + scheduler = again.schedulers[stub_account] + await _until(lambda: len(scheduler.items) == 1) + item = next(iter(scheduler.items.values())) + assert (item.msg_id, item.due_at) == (10, due) + status = again.v1_status() + assert status["actions"][stub_account]["pending"] == 1 + assert status["jobs"][0]["actions"][0]["pending"] == 1 + finally: + again.request_shutdown() + await asyncio.wait_for(runner, timeout=10) + + async def test_a_broken_edit_keeps_the_running_job( + self, tlgr_home: Path, stub_account: str, world + ): + from tlgr.daemon.app import Daemon + + (tlgr_home / "jobs.yaml").write_text(JOBS.format(account=stub_account)) + daemon = Daemon(tlgr_home, client_factory=fake_client_factory(world)) + runner = asyncio.create_task(daemon.run()) + try: + await _until(lambda: any(job.get("running") for job in daemon.list_jobs())) + (tlgr_home / "jobs.yaml").write_text( + JOBS.format(account=stub_account).replace("to: [", "too: [") + ) + result = await daemon.reload_jobs() + assert result["removed"] == [] + assert any("unknown key" in problem for problem in result["problems"]) + assert {job["name"] for job in daemon.list_jobs()} == {"archive"} + finally: + daemon.request_shutdown() + await asyncio.wait_for(runner, timeout=10) diff --git a/tests/test_filters_dialog.py b/tests/test_filters_dialog.py new file mode 100644 index 0000000..a08852f --- /dev/null +++ b/tests/test_filters_dialog.py @@ -0,0 +1,124 @@ +"""`sender_is_contact` and `chat_is_new`, on events built from real TL updates.""" + +from __future__ import annotations + +import pytest +from fake_telethon import FakeTelegramClient, World, make_user +from telethon.tl import types + +from tlgr.filters import dialog +from tlgr.filters.compose import evaluate, evaluate_async, parse_filter_config +from tlgr.gateway.engine import _builders +from tlgr.gateway.event import Event +from tlgr.gateway.tlevents import build_event + +ALICE = 4242 + + +@pytest.fixture(autouse=True) +def _fresh_caches(): + dialog.clear_caches() + yield + dialog.clear_caches() + + +def _private(message_id: int, *, sender=None) -> types.UpdateNewMessage: + message = types.Message( + id=message_id, + peer_id=types.PeerUser(user_id=ALICE), + from_id=types.PeerUser(user_id=ALICE), + date=None, + message="hello", + ) + update = types.UpdateNewMessage(message=message, pts=1, pts_count=1) + if sender is not None: + update._entities = {ALICE: sender} + return update + + +async def _event(client, update) -> Event: + built = await build_event(_builders(["new_message"]), update, client) + assert built is not None + return Event(source="telegram", raw=built[0], account="work") + + +class TestSenderIsContact: + async def test_a_contact_matches(self): + client = FakeTelegramClient(World()) + alice = make_user(ALICE) + alice.contact = True + event = await _event(client, _private(10, sender=alice)) + node = parse_filter_config({"sender_is_contact": True}) + assert (await evaluate_async(node, event))[0] is True + + async def test_a_stranger_does_not(self): + client = FakeTelegramClient(World()) + event = await _event(client, _private(10, sender=make_user(ALICE))) + ok, reason = await evaluate_async(parse_filter_config({"sender_is_contact": True}), event) + assert ok is False + assert "sender_is_contact=False" in reason + ok, _ = await evaluate_async(parse_filter_config({"sender_is_contact": False}), event) + assert ok is True + + async def test_the_answer_is_cached_for_an_update_without_the_entity(self): + client = FakeTelegramClient(World()) + alice = make_user(ALICE) + alice.contact = True + await evaluate_async( + parse_filter_config({"sender_is_contact": True}), + await _event(client, _private(10, sender=alice)), + ) + bare = await _event(client, _private(11)) + ok, _ = await evaluate_async(parse_filter_config({"sender_is_contact": True}), bare) + assert ok is True + + +class TestChatIsNew: + async def test_the_first_message_in_a_dialog_is_new(self): + world = World() + world.add_user(make_user(ALICE)) + client = FakeTelegramClient(world) + world.add_message(ALICE, "hello", message_id=500, sender_id=ALICE) + event = await _event(client, _private(500)) + assert (await evaluate_async(parse_filter_config({"chat_is_new": True}), event))[0] + + async def test_a_dialog_with_history_is_not_new(self): + world = World() + world.add_user(make_user(ALICE)) + client = FakeTelegramClient(world) + world.add_message(ALICE, "earlier", message_id=400, sender_id=ALICE) + world.add_message(ALICE, "hello", message_id=500, sender_id=ALICE) + event = await _event(client, _private(500)) + ok, reason = await evaluate_async(parse_filter_config({"chat_is_new": True}), event) + assert ok is False + assert "chat_is_new=False" in reason + + async def test_a_busy_dialog_costs_one_probe(self): + world = World() + world.add_user(make_user(ALICE)) + client = FakeTelegramClient(world) + probes = [] + original = client.get_messages + + async def counting(*args, **kwargs): + probes.append(kwargs) + return await original(*args, **kwargs) + + client.get_messages = counting + world.add_message(ALICE, "first", message_id=500, sender_id=ALICE) + node = parse_filter_config({"chat_is_new": True}) + assert (await evaluate_async(node, await _event(client, _private(500))))[0] is True + for message_id in (501, 502, 503): + world.add_message(ALICE, "more", message_id=message_id, sender_id=ALICE) + ok, _ = await evaluate_async(node, await _event(client, _private(message_id))) + assert ok is False + assert len(probes) == 1 + + +class TestSyncEvaluation: + async def test_an_async_filter_rejects_with_a_reason_instead_of_leaking(self): + client = FakeTelegramClient(World()) + event = await _event(client, _private(10, sender=make_user(ALICE))) + ok, reason = evaluate(parse_filter_config({"chat_is_new": True}), event) + assert ok is False + assert "needs the job engine" in reason diff --git a/tests/test_gateway.py b/tests/test_gateway.py index 1bbd821..2f6ec36 100644 --- a/tests/test_gateway.py +++ b/tests/test_gateway.py @@ -1,188 +1,180 @@ -"""End-to-end tests for the Gateway pipeline.""" +"""End-to-end tests for the Gateway pipeline: filters, processors, actions. + +The job is fed real TL updates and its actions land on a scheduler with a +virtual clock and a recording op runner, so each test reads what would have +reached the op layer. +""" from __future__ import annotations -from datetime import datetime, timezone -from unittest.mock import AsyncMock, MagicMock +from types import SimpleNamespace +from unittest.mock import AsyncMock import pytest +from fake_telethon import FakeTelegramClient, World +from job_helpers import ALICE, deliver, group_update, make_scheduler, private_update +from tlgr.actions import register_action from tlgr.filters.compose import parse_filter_config from tlgr.gateway.config import ActionConfig, GatewayConfig from tlgr.gateway.engine import Gateway from tlgr.processors import ProcessorChain -def _make_tg_event(text="hello", is_private=True, sender_id=100): - msg = MagicMock() - msg.text = text - msg.message = text - msg.sender_id = sender_id - msg.out = False - msg.reply_to = None - msg.forward = None - msg.media = None - msg.entities = None - msg.date = datetime(2025, 6, 15, 12, 0, 0, tzinfo=timezone.utc) - msg.action = None - msg.sender = MagicMock(bot=False) - - ev = MagicMock() - ev.message = msg - ev.chat_id = 42 - ev.is_private = is_private - ev.is_group = not is_private - ev.is_channel = False - ev.reply = AsyncMock() - - chat = MagicMock() - chat.title = "" - chat.megagroup = False - ev.chat = chat - - return ev - - -def _make_client(): - client = MagicMock() - client.resolve_chat = AsyncMock(return_value=999) - client.client = MagicMock() - client.client.forward_messages = AsyncMock() - client.client.send_message = AsyncMock() - client.client.send_file = AsyncMock() - client.client.on = MagicMock(side_effect=lambda *a, **kw: lambda f: f) - return client +def _client(): + return SimpleNamespace(client=FakeTelegramClient(World()), resolve_chat=AsyncMock()) + + +def _reply(text, **kwargs): + return ActionConfig(name="reply", config={"text": text, "typing": False}, **kwargs) + + +@pytest.fixture +async def sched(): + scheduler, runner, clock = make_scheduler() + scheduler.start() + try: + yield scheduler, runner, clock + finally: + await scheduler.stop(timeout=0.1) class TestGatewayPipeline: - @pytest.mark.asyncio - async def test_filter_match_triggers_action(self): + async def test_filter_match_triggers_action(self, sched): + scheduler, runner, clock = sched config = GatewayConfig( name="test-reply", - account="test", + account="work", filters=parse_filter_config({"chat_type": "private"}), - actions=[ActionConfig(name="reply", config="hello!")], + actions=[_reply("hello!")], ) - client = _make_client() - gw = Gateway(config, client) + gw = Gateway(config, _client(), scheduler=scheduler) await gw.setup() - tg_event = _make_tg_event(is_private=True) - await gw._handle(tg_event) + await deliver(gw, private_update(10)) + await clock.advance(1) - tg_event.reply.assert_awaited_once_with("hello!") + assert runner.requests("message.send") == [ + {"chat": str(ALICE), "text": "hello!", "reply_to": 10, "parse": "md"} + ] assert gw._stats["matched"] == 1 - @pytest.mark.asyncio - async def test_filter_mismatch_skips(self): + async def test_filter_mismatch_skips(self, sched): + scheduler, runner, clock = sched config = GatewayConfig( name="test-skip", - account="test", + account="work", filters=parse_filter_config({"chat_type": "private"}), - actions=[ActionConfig(name="reply", config="hello!")], + actions=[_reply("hello!")], ) - client = _make_client() - gw = Gateway(config, client) + gw = Gateway(config, _client(), scheduler=scheduler) await gw.setup() - tg_event = _make_tg_event(is_private=False) - await gw._handle(tg_event) + await deliver(gw, group_update(10)) + await clock.advance(1) - tg_event.reply.assert_not_awaited() + assert runner.calls == [] assert gw._stats["skipped"] == 1 - @pytest.mark.asyncio - async def test_no_filters_matches_all(self): - config = GatewayConfig( - name="test-all", - account="test", - filters=None, - actions=[ActionConfig(name="reply", config="yo")], - ) - client = _make_client() - gw = Gateway(config, client) - await gw.setup() + async def test_no_filters_matches_all(self, sched): + scheduler, runner, clock = sched + config = GatewayConfig(name="test-all", account="work", actions=[_reply("yo")]) + gw = Gateway(config, _client(), scheduler=scheduler) - tg_event = _make_tg_event() - await gw._handle(tg_event) + await deliver(gw, group_update(10)) + await clock.advance(1) - tg_event.reply.assert_awaited_once_with("yo") + assert runner.requests("message.send")[0]["text"] == "yo" - @pytest.mark.asyncio - async def test_multiple_actions(self): + async def test_multiple_actions(self, sched): + scheduler, runner, clock = sched config = GatewayConfig( name="test-multi", - account="test", - actions=[ - ActionConfig(name="reply", config="got it!"), - ActionConfig(name="reply", config="second reply"), - ], + account="work", + actions=[_reply("got it!"), _reply("second reply")], ) - client = _make_client() - gw = Gateway(config, client) - await gw.setup() + gw = Gateway(config, _client(), scheduler=scheduler) - tg_event = _make_tg_event() - await gw._handle(tg_event) + await deliver(gw, private_update(10)) + await clock.run_for(5) - assert tg_event.reply.await_count == 2 + assert [r["text"] for r in runner.requests("message.send")] == ["got it!", "second reply"] - @pytest.mark.asyncio - async def test_per_action_filter(self): + async def test_per_action_filter(self, sched): + scheduler, runner, clock = sched config = GatewayConfig( name="test-per-action", - account="test", + account="work", actions=[ - ActionConfig( - name="reply", - config="private only", - filters=parse_filter_config({"chat_type": "private"}), - ), - ActionConfig(name="reply", config="always"), + _reply("private only", filters=parse_filter_config({"chat_type": "private"})), + _reply("always"), ], ) - client = _make_client() - gw = Gateway(config, client) - await gw.setup() + gw = Gateway(config, _client(), scheduler=scheduler) - tg_event = _make_tg_event(is_private=False) - await gw._handle(tg_event) + await deliver(gw, group_update(10)) + await clock.advance(1) - # First action should be skipped (filter mismatch), second should run - assert tg_event.reply.await_count == 1 - assert tg_event.reply.call_args[0][0] == "always" + assert [r["text"] for r in runner.requests("message.send")] == ["always"] - @pytest.mark.asyncio - async def test_job_level_processors(self): + async def test_job_level_processors(self, sched): + scheduler, runner, clock = sched chain = ProcessorChain().add("add_prefix", {"prefix": "[BOT]"}) config = GatewayConfig( - name="test-proc", - account="test", - processors=chain, - actions=[ActionConfig(name="reply", config="hello")], + name="test-proc", account="work", processors=chain, actions=[_reply("hello")] ) - client = _make_client() - gw = Gateway(config, client) - await gw.setup() + gw = Gateway(config, _client(), scheduler=scheduler) - tg_event = _make_tg_event() - await gw._handle(tg_event) + await deliver(gw, private_update(10)) + await clock.advance(1) - call_text = tg_event.reply.call_args[0][0] - assert "[BOT]" in call_text + assert "[BOT]" in runner.requests("message.send")[0]["text"] - @pytest.mark.asyncio - async def test_unknown_action_logs_error(self): + async def test_unknown_action_logs_error(self, sched): + scheduler, runner, clock = sched config = GatewayConfig( name="test-unknown", - account="test", + account="work", actions=[ActionConfig(name="nonexistent_action", config="x")], ) - client = _make_client() - gw = Gateway(config, client) - await gw.setup() + gw = Gateway(config, _client(), scheduler=scheduler) + + await deliver(gw, private_update(10)) - tg_event = _make_tg_event() - await gw._handle(tg_event) + assert gw._stats["errors"] == 1 + + async def test_a_function_action_still_runs_at_once(self, sched): + scheduler, runner, clock = sched + seen = [] + + @register_action("_test_legacy") + async def legacy(event, config, client, chain=None): + seen.append((event.raw.message.id, config)) + config = GatewayConfig( + name="test-legacy", + account="work", + actions=[ActionConfig(name="_test_legacy", config="cfg")], + ) + gw = Gateway(config, _client(), scheduler=scheduler) + await deliver(gw, private_update(10)) + assert seen == [(10, "cfg")] + + async def test_status_reports_per_action_counters(self, sched): + scheduler, runner, clock = sched + config = GatewayConfig(name="test-status", account="work", actions=[_reply("hi")]) + gw = Gateway(config, _client(), scheduler=scheduler) + await deliver(gw, private_update(10)) + status = gw.status() + assert status["matched"] == 1 + assert status["actions"][0]["pending"] == 1 + await clock.advance(1) + status = gw.status() + assert status["actions"][0]["done"] == 1 + assert status["actions"][0]["pending"] == 0 + + async def test_without_a_scheduler_an_action_is_an_error_not_a_crash(self): + config = GatewayConfig(name="test-none", account="work", actions=[_reply("hi")]) + gw = Gateway(config, _client()) + await deliver(gw, private_update(10)) assert gw._stats["errors"] == 1 diff --git a/tests/test_gateway_bus.py b/tests/test_gateway_bus.py index b3735f2..393e15c 100644 --- a/tests/test_gateway_bus.py +++ b/tests/test_gateway_bus.py @@ -1,10 +1,11 @@ """A gateway job fed from the daemon's event bus, with real TL updates. The bus hands a job the raw TL `Update*` beside the envelope, while every -filter and action reads a high-level Telethon event (`event.chat_id`, -`event.is_private`, `event.reply()`). These tests drive a job the way the -daemon does, so a shape mismatch fails here instead of in production, where -it showed up as `matched=0` for days. +filter reads a high-level Telethon event (`event.chat_id`, `event.is_private`). +These tests drive a job the way the daemon does, and its actions run as real +operations through the daemon's dispatcher against the fake client, so a +shape mismatch anywhere between the update and the request fails here instead +of in production, where it showed up as `matched=0` for days. """ from __future__ import annotations @@ -14,7 +15,8 @@ from unittest.mock import AsyncMock import pytest -from fake_telethon import FakeTelegramClient, World +from fake_telethon import make_channel, make_user +from job_helpers import ALICE, daemon_job_env, photo_media, until from telethon.tl import types from tlgr.gateway.config import _parse_job @@ -22,7 +24,8 @@ SOURCE = 1234 SOURCE_MARKED = -1000000001234 -ALICE = 4242 +ARCHIVE = 9999 +ARCHIVE_MARKED = -1000000009999 class _Bus: @@ -35,30 +38,27 @@ def add_handler(self, handler) -> None: def remove_handler(self, handler) -> None: self.handlers.remove(handler) - async def deliver(self, update, *, event_type: str = "message_new", account: str = "Neo"): + async def deliver(self, update, *, event_type: str = "message_new", account: str = "work"): envelope = SimpleNamespace(type=event_type, account=account) for handler in list(self.handlers): await handler(envelope, update) -class _JobClient: - def __init__(self, world: World, chats: dict[str, int]) -> None: - self.client = FakeTelegramClient(world) - self.client.forward_messages = AsyncMock() - self.client.send_message = AsyncMock() - self.resolve_chat = AsyncMock(side_effect=lambda ref: chats[ref]) - - -def _channel_post(text: str = "breaking", *, channel: int = SOURCE, out: bool = False): +def _channel_post(text: str = "breaking", *, channel: int = SOURCE, msg_id: int = 5, media=None): message = types.Message( - id=5, peer_id=types.PeerChannel(channel_id=channel), date=None, message=text, out=out + id=msg_id, + peer_id=types.PeerChannel(channel_id=channel), + date=None, + message=text, + post=True, + media=media, ) return types.UpdateNewChannelMessage(message=message, pts=1, pts_count=1) -def _private(text: str = "hi", *, out: bool = False): +def _private(text: str = "hi", *, out: bool = False, msg_id: int = 6): message = types.Message( - id=6, + id=msg_id, peer_id=types.PeerUser(user_id=ALICE), from_id=None if out else types.PeerUser(user_id=ALICE), date=None, @@ -83,88 +83,111 @@ def bus() -> _Bus: @pytest.fixture -async def client() -> _JobClient: - return _JobClient(World(), {"@source": SOURCE_MARKED, "@archive": -1000000009999}) +async def env(daemon, world): + world.add_user(make_user(ALICE, username="alice")) + world.add_channel(make_channel(SOURCE, title="Source")) + world.add_channel(make_channel(ARCHIVE, title="Archive")) + world.add_message(SOURCE_MARKED, "breaking", message_id=5) + world.add_message(ALICE, "hi", message_id=6, sender_id=ALICE) + env = await daemon_job_env(daemon) + env.client = SimpleNamespace( + client=env.client.client, + resolve_chat=AsyncMock( + side_effect=lambda ref: {"@source": SOURCE_MARKED, "@archive": ARCHIVE_MARKED}[ref] + ), + ) + return env + + +def _forward_job(**action): + return _parse_job( + { + "name": "archive", + "account": "work", + "filters": {"chat_id": "@source"}, + "actions": [{"forward": {"to": [str(ARCHIVE_MARKED)], **action}}], + } + ) class TestForwardFromTheBus: - async def test_a_post_in_the_watched_channel_is_forwarded(self, bus, client): - config = _parse_job( - { - "name": "archive", - "account": "Neo", - "filters": {"chat_id": "@source"}, - "actions": [{"forward": {"to": ["@archive"]}}], - } - ) - job = Gateway(config, client, bus=bus) + async def test_a_post_in_the_watched_channel_is_forwarded(self, bus, env, world): + job = Gateway(_forward_job(), env.client, bus=bus, scheduler=env.scheduler) await _running(job, bus) try: await bus.deliver(_channel_post()) + await until(lambda: world.called("ForwardMessagesRequest"), clock=env.clock) finally: await job.stop() assert job._stats == {"matched": 1, "skipped": 0, "errors": 0} - client.client.forward_messages.assert_awaited_once() - assert client.client.forward_messages.await_args.args[0] == -1000000009999 + sent = world.called("ForwardMessagesRequest")[0] + assert list(sent.id) == [5] + assert sent.drop_author is None + assert env.scheduler.job_stats("archive")[0]["done"] == 1 - async def test_a_post_elsewhere_is_skipped(self, bus, client): - config = _parse_job( - { - "name": "archive", - "account": "Neo", - "filters": {"chat_id": "@source"}, - "actions": [{"forward": {"to": ["@archive"]}}], - } - ) - job = Gateway(config, client, bus=bus) + async def test_drop_author_reaches_the_request(self, bus, env, world): + job = Gateway(_forward_job(drop_author=True), env.client, bus=bus, scheduler=env.scheduler) + await _running(job, bus) + try: + await bus.deliver(_channel_post()) + await until(lambda: world.called("ForwardMessagesRequest"), clock=env.clock) + finally: + await job.stop() + assert world.called("ForwardMessagesRequest")[0].drop_author is True + + async def test_a_post_elsewhere_is_skipped(self, bus, env, world): + job = Gateway(_forward_job(), env.client, bus=bus, scheduler=env.scheduler) await _running(job, bus) try: await bus.deliver(_channel_post(channel=777)) + await env.clock.advance(5) finally: await job.stop() assert job._stats["matched"] == 0 assert job._stats["skipped"] == 1 - client.client.forward_messages.assert_not_awaited() - - async def test_the_processed_copy_is_sent_with_the_rewritten_text(self, bus, client): - config = _parse_job( - { - "name": "rename", - "account": "Neo", - "filters": {"chat_id": "@source"}, - "actions": [ - { - "forward": { - "to": ["@archive"], - "processors": [ - {"type": "regex", "pattern": "wbnet", "replacement": "VPEON"} - ], - } - } - ], - } + assert not world.called("ForwardMessagesRequest") + + async def test_the_processed_copy_is_sent_with_the_rewritten_text(self, bus, env, world): + config = _forward_job( + processors=[ + {"type": "regex", "pattern": "wbnet", "replacement": "VPEON", "flags": "i"} + ], + drop_author=True, ) - job = Gateway(config, client, bus=bus) + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) await _running(job, bus) try: - await bus.deliver(_channel_post("join wbnet today")) + await bus.deliver(_channel_post("join WBNET today")) + await until(lambda: world.called("SendMessageRequest"), clock=env.clock) finally: await job.stop() - client.client.send_message.assert_awaited_once_with(-1000000009999, "join VPEON today") + sent = world.called("SendMessageRequest")[0] + assert sent.message == "join VPEON today" + assert not world.called("ForwardMessagesRequest") + assert world.history(ARCHIVE_MARKED)[-1].message == "join VPEON today" - async def test_another_accounts_post_is_ignored(self, bus, client): - config = _parse_job( - { - "name": "archive", - "account": "Neo", - "filters": {"chat_id": "@source"}, - "actions": [{"forward": {"to": ["@archive"]}}], - } + async def test_processed_media_is_re_sent_with_the_rewritten_caption(self, bus, env, world): + world.add_media_message(SOURCE_MARKED, message_id=8, text="wbnet photo") + config = _forward_job( + processors=[{"type": "regex", "pattern": "wbnet", "replacement": "VPEON"}] ) - job = Gateway(config, client, bus=bus) + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) + await _running(job, bus) + try: + await bus.deliver(_channel_post("wbnet photo", msg_id=8, media=photo_media())) + await until(lambda: world.called("SendMediaRequest"), clock=env.clock) + finally: + await job.stop() + + sent = world.called("SendMediaRequest")[0] + assert sent.message == "VPEON photo" + assert type(sent.media).__name__ == "InputMediaPhoto" + + async def test_another_accounts_post_is_ignored(self, bus, env, world): + job = Gateway(_forward_job(), env.client, bus=bus, scheduler=env.scheduler) await _running(job, bus) try: await bus.deliver(_channel_post(), account="other") @@ -180,95 +203,104 @@ def config(self): return _parse_job( { "name": "away", - "account": "Neo", + "account": "work", "filters": {"chat_type": "private"}, "actions": [{"reply": "away"}], } ) - async def test_an_incoming_private_message_is_answered(self, bus, client, config): - client.client.send_message = AsyncMock() - job = Gateway(config, client, bus=bus) + async def test_an_incoming_private_message_is_answered(self, bus, env, world, config): + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) await _running(job, bus) try: await bus.deliver(_private()) + await until(lambda: world.called("SendMessageRequest"), clock=env.clock) finally: await job.stop() assert job._stats["matched"] == 1 - client.client.send_message.assert_awaited_once() + sent = world.called("SendMessageRequest")[0] + assert sent.message == "away" + assert sent.reply_to.reply_to_msg_id == 6 + # "typing..." first, by default. + assert world.called("SetTypingRequest") - async def test_the_accounts_own_message_is_not_answered(self, bus, client, config): + async def test_the_accounts_own_message_is_not_answered(self, bus, env, world, config): """`new_message` means incoming, as it did when jobs used `NewMessage(incoming=True)`.""" - job = Gateway(config, client, bus=bus) + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) await _running(job, bus) try: await bus.deliver(_private(out=True)) + await env.clock.advance(20) finally: await job.stop() assert job._stats == {"matched": 0, "skipped": 0, "errors": 0} - client.client.send_message.assert_not_awaited() + assert not world.called("SendMessageRequest") -class TestChatRefResolution: - async def test_numeric_refs_are_left_alone(self, bus, client): - config = _parse_job( - { - "name": "ids", - "account": "Neo", - "filters": {"chat_id": [SOURCE_MARKED, "@source"]}, - "actions": [{"forward": {"to": ["@archive"]}}], - } - ) - job = Gateway(config, client, bus=bus) - await job.setup() - - assert config.filters.filter_value == [SOURCE_MARKED, SOURCE_MARKED] - client.resolve_chat.assert_awaited_once_with("@source") - - async def test_an_unresolvable_ref_is_logged_and_kept(self, bus, client, caplog): - client.resolve_chat = AsyncMock(side_effect=ValueError("no such chat")) - config = _parse_job( - { - "name": "typo", - "account": "Neo", - "filters": {"chat_id": "@nope"}, - "actions": [{"forward": {"to": ["@archive"]}}], - } +class TestReactReadViewEndToEnd: + async def test_react_reads_first_then_replaces_the_reaction(self, bus, env, world): + config = _parse_job({"name": "ack", "account": "work", "actions": [{"react": "👍"}]}) + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) + await _running(job, bus) + try: + await bus.deliver(_private()) + await until(lambda: world.called("SendReactionRequest"), clock=env.clock) + finally: + await job.stop() + names = [name for name, _ in world.calls] + assert names.index("ReadHistoryRequest") < names.index("SendReactionRequest") + assert world.read_inbox[ALICE] == 6 + reaction = world.called("SendReactionRequest")[0] + assert [r.emoticon for r in reaction.reaction] == ["👍"] + + async def test_the_read_rpc_follows_the_peer(self, bus, env, world): + config = _parse_job({"name": "reader", "account": "work", "actions": [{"read": {}}]}) + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) + await _running(job, bus) + try: + await bus.deliver(_private()) + await bus.deliver(_channel_post()) + await until(lambda: len(world.called("ReadHistoryRequest")) == 2, clock=env.clock) + finally: + await job.stop() + private, channel = world.called("ReadHistoryRequest") + assert type(private).__module__.endswith("messages") + assert private.max_id == 6 + assert type(channel).__module__.endswith("channels") + assert channel.max_id == 5 + + async def test_a_forum_topic_is_read_with_read_discussion(self, bus, env, world): + world.add_channel(make_channel(4444, title="Forum", megagroup=True)) + config = _parse_job({"name": "reader", "account": "work", "actions": [{"read": {}}]}) + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) + message = types.Message( + id=70, + peer_id=types.PeerChannel(channel_id=4444), + from_id=types.PeerUser(user_id=ALICE), + date=None, + message="topic chatter", + reply_to=types.MessageReplyHeader(reply_to_msg_id=60, forum_topic=True), ) - job = Gateway(config, client, bus=bus) - await job.setup() - - assert config.filters.filter_value == "@nope" - assert "cannot resolve chat @nope" in caplog.text - - async def test_a_ref_that_failed_at_boot_is_retried_when_events_arrive(self, bus, client): - """An account still offline when the job starts must not leave it deaf for good.""" - answers = [ValueError("offline"), SOURCE_MARKED] - - async def resolve(ref): - answer = answers.pop(0) - if isinstance(answer, Exception): - raise answer - return answer + await _running(job, bus) + try: + await bus.deliver(types.UpdateNewChannelMessage(message=message, pts=1, pts_count=1)) + await until(lambda: world.called("ReadDiscussionRequest"), clock=env.clock) + finally: + await job.stop() + sent = world.called("ReadDiscussionRequest")[0] + assert (sent.msg_id, sent.read_max_id) == (60, 70) - client.resolve_chat = AsyncMock(side_effect=resolve) - config = _parse_job( - { - "name": "archive", - "account": "Neo", - "filters": {"chat_id": "@source"}, - "actions": [{"forward": {"to": [-1000000009999]}}], - } - ) - job = Gateway(config, client, bus=bus) + async def test_a_channel_post_view_is_incremented(self, bus, env, world): + config = _parse_job({"name": "viewer", "account": "work", "actions": [{"view": {}}]}) + job = Gateway(config, env.client, bus=bus, scheduler=env.scheduler) await _running(job, bus) try: - job._next_resolve = 0.0 # the retry interval has passed await bus.deliver(_channel_post()) + await until(lambda: world.called("GetMessagesViewsRequest"), clock=env.clock) finally: await job.stop() - - assert job._stats["matched"] == 1 - assert config.filters.filter_value == SOURCE_MARKED + sent = world.called("GetMessagesViewsRequest")[0] + assert sent.increment is True + assert list(sent.id) == [5] diff --git a/tests/test_gateway_config.py b/tests/test_gateway_config.py new file mode 100644 index 0000000..995b3ee --- /dev/null +++ b/tests/test_gateway_config.py @@ -0,0 +1,197 @@ +"""`jobs.yaml`: the action syntax, the shared knobs, `pacing:`, and the errors.""" + +from __future__ import annotations + +import pytest +import yaml + +from tlgr.gateway.config import JobConfigError, _parse_job, parse_jobs_document +from tlgr.gateway.knobs import ActionKnobs, merge_knobs +from tlgr.gateway.pacer import DEFAULT_PACING, PacingRule + +#: The shape of the forward jobs running in production; they must load as before. +PRODUCTION = """\ +jobs: +- name: ircfspace-to-vpeon + account: Neo + filters: + chat_id: '@ircfspace' + actions: + - forward: + to: + - '@VPEON' +- name: wbnet-to-vpeon + account: Neo + filters: + chat_id: '@wbnet' + actions: + - forward: + to: + - '@VPEON' + drop_author: true + processors: + - type: regex + pattern: wbnet + replacement: VPEON + flags: i +""" + +APPROVED = """\ +pacing: + Neo: + react: {every: 4s, per_hour: 300} + read: {every: 2s} + view: {every: 2s} + forward: {every: 1.5s} + reply: {every: 1.5s} + expire: {react: 24h, view: 24h, reply: 24h} + +jobs: + - name: dm-ack + account: Neo + filters: {chat_type: private, sender_is_contact: true} + presence: {mode: session, quiet_hours: "01:00-08:00"} + on_takeover: cancel + actions: + - read: {delay: 10-90s} + - view: {delay: 15-120s} + - react: + emoji: ["👍", "❤", "🔥"] + percent: 60 + delay: 30-300s + - reply: + text: "Got it, will answer soon" + filters: {chat_is_new: true} + typing: true + delay: 1-3m + + - name: wbnet-to-vpeon + account: Neo + filters: {chat_id: "@wbnet"} + actions: + - forward: {to: ["@VPEON"], drop_author: true} +""" + + +class TestProductionJobs: + def test_the_running_forward_jobs_load_with_their_old_meaning(self): + loaded = parse_jobs_document(yaml.safe_load(PRODUCTION)) + assert loaded.problems == [] + plain, rewritten = loaded.jobs + assert plain.actions[0].params == {"to": ["@VPEON"], "drop_author": False} + assert plain.actions[0].processors is None + assert rewritten.actions[0].params == {"to": ["@VPEON"], "drop_author": True} + assert rewritten.actions[0].processors.apply("join WBNET") == "join VPEON" + # No knobs: act at once, every time, presence untouched. + assert merge_knobs(rewritten.knobs, rewritten.actions[0].knobs) == ActionKnobs() + + +class TestApprovedSyntax: + def test_the_example_the_user_approved_parses(self): + loaded = parse_jobs_document(yaml.safe_load(APPROVED)) + assert loaded.problems == [] + dm = loaded.jobs[0] + assert dm.knobs["presence"].mode == "session" + assert dm.knobs["presence"].quiet_hours.text == "01:00-08:00" + read, view, react, reply = dm.actions + assert read.knobs["delay"] == (10.0, 90.0) + assert react.knobs["percent"] == 60 + assert react.params["choices"] == [["👍", 1.0], ["❤", 1.0], ["🔥", 1.0]] + assert reply.params == {"text": "Got it, will answer soon", "typing": True} + assert reply.knobs["delay"] == (60.0, 180.0) + assert reply.filters is not None + + pacing = loaded.pacing["Neo"] + assert pacing.rules["react"] == PacingRule(every=4.0, per_hour=300) + assert pacing.rules["forward"] == PacingRule(every=1.5, per_hour=None) + assert pacing.expire == {"react": 86400.0, "view": 86400.0, "reply": 86400.0} + + def test_short_forms(self): + job = _parse_job( + { + "name": "short", + "actions": [{"read": {}}, {"read": True}, {"view": None}, {"reply": "hi"}], + } + ) + assert [a.name for a in job.actions] == ["read", "read", "view", "reply"] + + def test_pacing_keeps_the_default_cap_unless_lifted(self): + loaded = parse_jobs_document( + {"pacing": {"Neo": {"react": {"every": "6s"}, "view": {"per_hour": None}}}} + ) + assert loaded.pacing["Neo"].rules["react"] == PacingRule(every=6.0, per_hour=300) + assert loaded.pacing["Neo"].rules["view"].every == DEFAULT_PACING["view"].every + + def test_expire_never(self): + loaded = parse_jobs_document({"pacing": {"Neo": {"expire": {"react": "never"}}}}) + assert loaded.pacing["Neo"].expire == {"react": None} + + +class TestErrors: + @pytest.mark.parametrize( + ("job", "fragment"), + [ + ({"name": "x", "actions": [{"read": {"delay": "soonish"}}]}, "delay"), + ({"name": "x", "actions": [{"react": {"emoji": "👍", "percent": 101}}]}, "percent"), + ({"name": "x", "presence": "always", "actions": [{"read": {}}]}, "presence"), + ({"name": "x", "on_takeover": "panic", "actions": [{"read": {}}]}, "on_takeover"), + ({"name": "x", "actions": [{"read": {"histroy": True}}]}, "unknown key"), + ({"name": "x", "colour": "red", "actions": [{"read": {}}]}, "unknown key"), + ({"name": "x", "actions": [{"teleport": {}}]}, "unknown action"), + ({"name": "x", "actions": [{"react": {"👍": 1, "delay": "5s"}}]}, "long form"), + ({"name": "x", "actions": [{"read": {"processors": ["x"]}}]}, "processors"), + ({"name": "x", "actions": [{"forward": {}}]}, "to"), + ({"actions": [{"read": {}}]}, "name"), + ], + ) + def test_each_problem_is_named(self, job, fragment): + with pytest.raises(JobConfigError) as caught: + _parse_job(job) + assert fragment in str(caught.value) + + def test_problems_carry_the_job_and_the_action_position(self): + with pytest.raises(JobConfigError) as caught: + _parse_job({"name": "dm", "actions": [{"read": {}}, {"react": {"emoji": ""}}]}) + assert "job 'dm', action 2" in str(caught.value) + + def test_a_broken_job_does_not_take_the_others_with_it(self): + loaded = parse_jobs_document( + { + "jobs": [ + {"name": "good", "actions": [{"forward": {"to": "@a"}}]}, + {"name": "bad", "actions": [{"read": {"delay": "x"}}]}, + ] + } + ) + assert [job.name for job in loaded.jobs] == ["good"] + assert loaded.rejected == ["bad"] + assert len(loaded.problems) == 1 + + @pytest.mark.parametrize( + "pacing", + [ + {"Neo": {"react": {"every": "x"}}}, + {"Neo": {"react": {"per_hour": 0}}}, + {"Neo": {"tickle": {"every": "1s"}}}, + {"Neo": {"react": {"every": "1s", "burst": 3}}}, + {"Neo": {"expire": {"read": "soon"}}}, + {"Neo": {"expire": {"poke": "1h"}}}, + ["Neo"], + ], + ) + def test_bad_pacing_is_reported(self, pacing): + assert parse_jobs_document({"pacing": pacing, "jobs": []}).problems + + def test_a_duplicate_name_is_reported(self): + loaded = parse_jobs_document({"jobs": [{"name": "a", "actions": [{"read": {}}]}] * 2}) + assert [job.name for job in loaded.jobs] == ["a"] + assert "used twice" in loaded.problems[0] + + +def test_the_shipped_example_validates(): + from pathlib import Path + + example = Path(__file__).resolve().parent.parent / "jobs.example.yaml" + loaded = parse_jobs_document(yaml.safe_load(example.read_text(encoding="utf-8"))) + assert loaded.problems == [] + assert "dm-ack" in {job.name for job in loaded.jobs} diff --git a/tests/test_gateway_knobs.py b/tests/test_gateway_knobs.py new file mode 100644 index 0000000..4dbac65 --- /dev/null +++ b/tests/test_gateway_knobs.py @@ -0,0 +1,129 @@ +"""Parsing the knobs every job action shares.""" + +from __future__ import annotations + +from datetime import datetime + +import pytest + +from tlgr.gateway.knobs import ( + ActionKnobs, + KnobError, + QuietHours, + merge_knobs, + parse_delay, + parse_duration, + parse_knobs, + parse_percent, + parse_presence, +) + + +class TestDurations: + @pytest.mark.parametrize( + ("raw", "seconds"), + [("5s", 5.0), ("500ms", 0.5), ("2m", 120.0), ("1.5s", 1.5), ("24h", 86400.0), (30, 30.0)], + ) + def test_units(self, raw, seconds): + assert parse_duration(raw) == seconds + + @pytest.mark.parametrize("raw", ["soon", "5 parsecs", "-3s", True, [], "1x"]) + def test_garbage_is_refused(self, raw): + with pytest.raises(KnobError): + parse_duration(raw) + + @pytest.mark.parametrize( + ("raw", "pair"), + [ + ("10-90s", (10.0, 90.0)), + ("1-3m", (60.0, 180.0)), + ("1m-3m", (60.0, 180.0)), + ("500ms-2s", (0.5, 2.0)), + ("5s", (5.0, 5.0)), + (None, (0.0, 0.0)), + ], + ) + def test_ranges(self, raw, pair): + assert parse_delay(raw) == pair + + def test_an_inverted_range_is_refused(self): + with pytest.raises(KnobError, match="low end above"): + parse_delay("90-10s") + + +class TestPercent: + @pytest.mark.parametrize("raw", [0, 60, 100, "60", "60%"]) + def test_valid(self, raw): + assert 0 <= parse_percent(raw) <= 100 + + @pytest.mark.parametrize("raw", [-1, 101, 50.5, "half", True]) + def test_invalid(self, raw): + with pytest.raises(KnobError): + parse_percent(raw) + + +class TestPresence: + def test_short_form(self): + assert parse_presence("session").mode == "session" + + def test_long_form_with_quiet_hours(self): + presence = parse_presence({"mode": "blip", "quiet_hours": "01:00-08:00"}) + assert presence.mode == "blip" + assert presence.quiet_hours == QuietHours(60, 480, "01:00-08:00") + + @pytest.mark.parametrize( + "raw", ["always", {"mode": "loud"}, {"mode": "blip", "quiet": "1-2"}, 7] + ) + def test_invalid(self, raw): + with pytest.raises(KnobError): + parse_presence(raw) + + +class TestQuietHours: + def test_a_window_inside_one_day(self): + quiet = QuietHours.parse("01:00-08:00") + assert quiet.contains(datetime(2026, 1, 1, 3, 0)) + assert not quiet.contains(datetime(2026, 1, 1, 8, 0)) + assert quiet.window_end(datetime(2026, 1, 1, 3, 0)) == datetime(2026, 1, 1, 8, 0) + + def test_a_window_across_midnight(self): + quiet = QuietHours.parse("23:00-07:00") + assert quiet.contains(datetime(2026, 1, 1, 23, 30)) + assert quiet.contains(datetime(2026, 1, 2, 6, 59)) + assert quiet.window_end(datetime(2026, 1, 1, 23, 30)) == datetime(2026, 1, 2, 7, 0) + + def test_outside_the_window_nothing_is_held(self): + quiet = QuietHours.parse("01:00-08:00") + moment = datetime(2026, 1, 1, 12, 0) + assert quiet.window_end(moment) == moment + + @pytest.mark.parametrize("raw", ["25:00-08:00", "1-8", "01:00", None]) + def test_invalid(self, raw): + with pytest.raises(KnobError): + QuietHours.parse(raw) + + +class TestMerging: + def test_defaults_reproduce_the_old_behaviour(self): + knobs = merge_knobs({}, {}) + assert knobs == ActionKnobs() + assert knobs.delay == (0.0, 0.0) + assert knobs.percent == 100 + assert knobs.presence.mode == "leave" + assert knobs.dry_run is False + + def test_the_action_wins_over_the_job(self): + job = parse_knobs({"delay": "10-20s", "percent": 50, "on_takeover": "ignore"}) + action = parse_knobs({"percent": 80}) + knobs = merge_knobs(job, action) + assert knobs.delay == (10.0, 20.0) + assert knobs.percent == 80 + assert knobs.on_takeover == "ignore" + + def test_errors_name_where_they_came_from(self): + with pytest.raises(KnobError, match="job 'x', action 1: on_takeover"): + parse_knobs({"on_takeover": "panic"}, where="job 'x', action 1") + + def test_dry_run_must_be_a_boolean(self): + with pytest.raises(KnobError, match="dry_run"): + parse_knobs({"dry_run": "yes"}) diff --git a/tests/test_gateway_pacer.py b/tests/test_gateway_pacer.py new file mode 100644 index 0000000..4ecd5ce --- /dev/null +++ b/tests/test_gateway_pacer.py @@ -0,0 +1,76 @@ +"""The pacer: a floor between actions, jitter that only lengthens it, an hourly cap.""" + +from __future__ import annotations + +import itertools +import random + +from tlgr.gateway.pacer import DEFAULT_PACING, MAX_SLOWDOWN, Pacer, PacingRule + + +def _drain(pacer: Pacer, count: int, start: float = 0.0) -> list[float]: + """Take *count* slots as fast as the pacer allows; return when each opened.""" + now = start + taken: list[float] = [] + for _ in range(count): + now += pacer.delay(now) + pacer.consume(now) + taken.append(now) + return taken + + +class TestSpacing: + def test_the_floor_is_never_broken_and_jitter_stays_in_bounds(self): + pacer = Pacer(PacingRule(every=4.0), rng=random.Random(7)) + taken = _drain(pacer, 200) + gaps = [b - a for a, b in itertools.pairwise(taken)] + assert min(gaps) >= 4.0 + assert max(gaps) <= 6.0 + # Jitter, not a metronome. + assert len({round(gap, 3) for gap in gaps}) > 50 + + def test_a_free_slot_opens_immediately(self): + pacer = Pacer(PacingRule(every=2.0)) + assert pacer.delay(100.0) == 0.0 + + def test_a_backlog_drains_at_the_configured_rate(self): + """Three hundred replayed messages are all acted on, none skipped, at the pace.""" + pacer = Pacer(PacingRule(every=2.0), rng=random.Random(1)) + taken = _drain(pacer, 300) + assert len(taken) == 300 + span = taken[-1] - taken[0] + assert 299 * 2.0 <= span <= 299 * 3.0 + + +class TestHourlyCap: + def test_the_cap_holds_the_next_slot_for_the_rest_of_the_hour(self): + pacer = Pacer(PacingRule(every=1.0, per_hour=10), rng=random.Random(3)) + taken = _drain(pacer, 11) + assert taken[10] >= taken[0] + 3600.0 + + def test_no_rolling_hour_ever_exceeds_the_cap(self): + pacer = Pacer(DEFAULT_PACING["react"], rng=random.Random(5)) + taken = _drain(pacer, 900) + for index, moment in enumerate(taken): + in_window = [t for t in taken[index:] if t < moment + 3600.0] + assert len(in_window) <= 300 + + +class TestFloodWait: + def test_a_flood_holds_the_queue_and_slows_it(self): + pacer = Pacer(PacingRule(every=2.0), rng=random.Random(2)) + pacer.consume(0.0) + pacer.penalize(1.0, 30.0) + assert pacer.delay(1.0) >= 30.0 + assert pacer.slowdown == 2.0 + now = 31.0 + pacer.consume(now) + assert pacer.next_at - now >= 4.0 + + def test_the_slowdown_is_capped_and_recovers(self): + pacer = Pacer(PacingRule(every=1.0)) + for _ in range(10): + pacer.penalize(0.0, 1.0) + assert pacer.slowdown == MAX_SLOWDOWN + pacer.delay(601.0) + assert pacer.slowdown == 1.0 diff --git a/tests/test_gateway_scheduler.py b/tests/test_gateway_scheduler.py new file mode 100644 index 0000000..86fe159 --- /dev/null +++ b/tests/test_gateway_scheduler.py @@ -0,0 +1,731 @@ +"""The job-action scheduler: delays, pacing, persistence, coalescing, takeover. + +Driven with a virtual clock and a recording op runner, so an hour of pacing +takes milliseconds and every assertion is about what would have reached the +op layer, and when. +""" + +from __future__ import annotations + +import asyncio +import itertools +import logging +import time +from datetime import datetime +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from fake_clock import FakeClock +from fake_telethon import FakeTelegramClient, World +from job_helpers import ( + ALICE, + CHANNEL_ID, + bus_envelope, + channel_update, + deliver, + make_scheduler, + photo_media, + private_update, + voice_media, +) +from telethon import errors +from telethon.tl import types + +from tlgr.actions.read import reading_seconds +from tlgr.actions.reply import typing_seconds +from tlgr.gateway.config import _parse_job +from tlgr.gateway.engine import Gateway +from tlgr.gateway.pacer import PacingRule +from tlgr.gateway.pending import PendingStore +from tlgr.gateway.scheduler import AccountPacing + + +def _job(scheduler, actions, **job_keys): + config = _parse_job({"name": "dm", "account": "work", "actions": actions, **job_keys}) + client = SimpleNamespace(client=FakeTelegramClient(World()), resolve_chat=AsyncMock()) + return Gateway(config, client, scheduler=scheduler) + + +@pytest.fixture +async def sched(): + scheduler, runner, clock = make_scheduler() + scheduler.start() + try: + yield scheduler, runner, clock + finally: + await scheduler.stop(timeout=0.1) + + +def _stats(scheduler, index=0, job="dm"): + return next(row for row in scheduler.job_stats(job) if row["index"] == index) + + +class TestDelay: + async def test_the_bus_handler_never_waits_for_the_delay(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {"delay": "90s"}}]) + started = time.monotonic() + await deliver(job, private_update(10)) + assert time.monotonic() - started < 1.0 + assert runner.calls == [] + assert len(scheduler.items) == 1 + + await clock.advance(89) + assert runner.calls == [] + await clock.advance(2) + assert runner.ops() == ["message.read"] + + async def test_the_delay_counts_from_arrival_not_the_message_date(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"reply": {"text": "ok", "typing": False, "delay": "30s"}}]) + update = private_update(10) + update.message.date = datetime(2020, 1, 1) + await deliver(job, update) + item = next(iter(scheduler.items.values())) + assert item.due_at == pytest.approx(clock.now() + 30) + + async def test_each_action_has_its_own_delay(self, sched): + scheduler, runner, clock = sched + job = _job( + scheduler, + [{"forward": {"to": "@archive"}}, {"react": {"emoji": "👍", "delay": "120s"}}], + ) + await deliver(job, private_update(10)) + await clock.advance(1) + assert runner.ops() == ["message.forward"] + await clock.advance(120) + assert runner.ops() == ["message.forward", "message.read", "reaction.add"] + + +class TestPercent: + async def test_rolls_are_independent_and_reproducible(self): + outcomes = [] + for _ in range(2): + scheduler, runner, clock = make_scheduler(seed=42) + job = _job(scheduler, [{"react": {"emoji": "👍", "percent": 50}}]) + for message_id in range(1, 41): + await deliver(job, private_update(message_id)) + outcomes.append(sorted(item.msg_id for item in scheduler.items.values())) + assert _stats(scheduler)["skipped"] == 40 - len(outcomes[-1]) + assert outcomes[0] == outcomes[1] + assert 8 < len(outcomes[0]) < 32 + + async def test_zero_percent_never_acts(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {"percent": 0}}]) + await deliver(job, private_update(10)) + assert scheduler.items == {} + assert _stats(scheduler)["skipped"] == 1 + + +class TestPacing: + async def test_reactions_keep_the_floor_and_drain_a_backlog(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"react": "👍"}]) + for message_id in range(1, 31): + await deliver(job, private_update(message_id, user=ALICE + message_id)) + await clock.run_for(200, step=1.0) + times = runner.times("reaction.add") + assert len(times) == 30 + gaps = [b - a for a, b in itertools.pairwise(times)] + assert min(gaps) >= 4.0 + assert max(gaps) <= 6.0 + 1.0 + assert _stats(scheduler)["done"] == 30 + + async def test_a_reaction_backlog_does_not_delay_forwards(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"react": "👍"}, {"forward": {"to": "@archive"}}]) + for message_id in range(1, 11): + await deliver(job, private_update(message_id, user=ALICE + message_id)) + await clock.run_for(20, step=0.5) + assert len(runner.requests("message.forward")) == 10 + assert len(runner.requests("reaction.add")) < 10 + + async def test_configured_pacing_applies(self, sched): + scheduler, runner, clock = sched + scheduler.configure(AccountPacing(rules={"forward": PacingRule(every=10.0)})) + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + for message_id in range(1, 4): + await deliver(job, private_update(message_id)) + await clock.run_for(25, step=0.5) + times = runner.times("message.forward") + assert len(times) == 3 + assert times[1] - times[0] >= 10.0 + + +class TestPersistence: + async def test_pending_items_survive_a_restart_with_their_due_time(self, tmp_path): + store = PendingStore(tmp_path / "pending.json") + clock = FakeClock() + first, runner, _ = make_scheduler(clock=clock, store=store) + first.start() + job = _job(first, [{"reply": {"text": "later", "typing": False, "delay": "100s"}}]) + await deliver(job, private_update(10)) + due = next(iter(first.items.values())).due_at + await clock.advance(30) + await first.stop(timeout=0.1) + assert (tmp_path / "pending.json").stat().st_mode & 0o777 == 0o600 + + second, runner2, _ = make_scheduler(clock=clock, store=store) + assert second.resume({"dm"}) == 1 + second.start() + try: + assert next(iter(second.items.values())).due_at == due + await clock.advance(69) + assert runner2.calls == [] + await clock.advance(2) + assert runner2.ops() == ["message.send"] + assert runner2.requests("message.send")[0]["text"] == "later" + finally: + await second.stop(timeout=0.1) + + async def test_a_replay_before_resume_does_not_schedule_a_copy(self, tmp_path): + store = PendingStore(tmp_path / "pending.json") + first, _, clock = make_scheduler(store=store) + job = _job(first, [{"read": {"delay": "100s"}}]) + await deliver(job, private_update(10)) + first.flush() + + second, _, _ = make_scheduler(clock=clock, store=store) + second.preload() + await deliver(_job(second, [{"read": {"delay": "100s"}}]), private_update(10)) + assert second.items == {} + assert second.resume({"dm"}) == 1 + assert len(second.items) == 1 + + async def test_items_of_a_removed_job_are_dropped_on_resume(self, tmp_path): + store = PendingStore(tmp_path / "pending.json") + first, _, clock = make_scheduler(store=store) + job = _job(first, [{"read": {"delay": "100s"}}]) + await deliver(job, private_update(10)) + first.flush() + second, _, _ = make_scheduler(clock=clock, store=store) + assert second.resume({"another-job"}) == 0 + assert second.items == {} + + async def test_a_replayed_update_is_not_acted_on_twice(self, tmp_path): + store = PendingStore(tmp_path / "pending.json") + first, runner, clock = make_scheduler(store=store) + first.start() + job = _job(first, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await clock.advance(1) + await first.stop(timeout=0.1) + + second, runner2, _ = make_scheduler(clock=clock, store=store) + second.resume({"dm"}) + second.start() + try: + await deliver(_job(second, [{"forward": {"to": "@archive"}}]), private_update(10)) + await clock.advance(5) + assert runner2.calls == [] + finally: + await second.stop(timeout=0.1) + + +class TestExpiry: + async def test_a_reaction_expires_after_a_day_but_a_read_does_not(self, sched): + scheduler, runner, clock = sched + job = _job( + scheduler, + [{"react": {"emoji": "👍", "delay": "25h"}}, {"read": {"delay": "25h"}}], + ) + await deliver(job, private_update(10)) + await clock.advance(25 * 3600 + 120) + assert _stats(scheduler, 0)["expired"] == 1 + assert _stats(scheduler, 1)["done"] == 1 + assert runner.ops() == ["message.read"] + + async def test_expiry_is_configurable(self, sched): + scheduler, runner, clock = sched + scheduler.configure(AccountPacing(expire={"forward": 60.0})) + job = _job(scheduler, [{"forward": {"to": "@archive", "delay": "2m"}}]) + await deliver(job, private_update(10)) + await clock.advance(200) + assert _stats(scheduler)["expired"] == 1 + + +class TestReact: + async def test_the_reaction_replaces_and_follows_a_read_up_to_the_message(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"react": {"emoji": "🔥", "big": True}}]) + await deliver(job, private_update(10)) + await clock.advance(1) + assert runner.ops() == ["message.read", "reaction.add"] + assert runner.requests("message.read")[0] == {"chat": str(ALICE), "up_to": 10} + assert runner.requests("reaction.add")[0] == { + "chat": str(ALICE), + "msg_id": 10, + "emoji": ["🔥"], + "replace": True, + "big": True, + } + + async def test_a_pending_read_is_coalesced_into_the_implied_one(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {"delay": "10m"}}, {"react": "👍"}]) + await deliver(job, private_update(10)) + await clock.run_for(15 * 60, step=5) + assert runner.ops() == ["message.read", "reaction.add"] + assert _stats(scheduler, 0)["done"] == 1 + + async def test_no_read_when_the_chat_is_already_read_that_far(self, sched): + scheduler, runner, clock = sched + scheduler.external_read(ALICE, None, 50) + job = _job(scheduler, [{"react": "👍"}], on_takeover="ignore") + await deliver(job, private_update(10)) + await clock.advance(1) + assert runner.ops() == ["reaction.add"] + + async def test_random_and_weighted_picks(self): + scheduler, runner, clock = make_scheduler(seed=3) + job = _job(scheduler, [{"react": ["👍", "❤", "🔥"]}]) + for message_id in range(1, 61): + await deliver(job, private_update(message_id, user=ALICE + message_id)) + picked = {item.payload["emoji"] for item in scheduler.items.values()} + assert picked == {"👍", "❤", "🔥"} + + scheduler, runner, clock = make_scheduler(seed=3) + job = _job(scheduler, [{"react": {"👍": 9, "🔥": 1}}]) + for message_id in range(1, 201): + await deliver(job, private_update(message_id, user=ALICE + message_id)) + counts = [item.payload["emoji"] for item in scheduler.items.values()] + assert counts.count("👍") > 5 * counts.count("🔥") + + async def test_an_album_gets_one_reaction_on_its_caption(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"react": {"emoji": "👍", "delay": "5s"}}]) + await deliver(job, private_update(20, "", media=photo_media(), grouped_id=900)) + await deliver(job, private_update(21, "look", media=photo_media(), grouped_id=900)) + await deliver(job, private_update(22, "", media=photo_media(), grouped_id=900)) + await clock.advance(10) + reactions = runner.requests("reaction.add") + assert len(reactions) == 1 + assert reactions[0]["msg_id"] == 21 + + async def test_reaction_invalid_is_an_error_and_is_not_retried(self, sched): + scheduler, runner, clock = sched + runner.fail("reaction.add", errors.ReactionInvalidError(request=None)) + job = _job(scheduler, [{"react": "🦄"}]) + await deliver(job, private_update(10)) + await clock.advance(200) + assert len(runner.requests("reaction.add")) == 1 + stats = _stats(scheduler) + assert stats["errors"] == 1 + assert "REACTION_INVALID" in (stats["last_error"] or "").upper() + + +class TestRead: + async def test_due_reads_coalesce_to_the_highest_id(self, sched): + scheduler, runner, clock = sched + quick = _job(scheduler, [{"read": {}}]) + await deliver(quick, private_update(10, "")) + await deliver(quick, private_update(12, "")) + slow = _job(scheduler, [{"read": {"delay": "100s"}}]) + await deliver(slow, private_update(15, "")) + await clock.advance(1) + assert runner.requests("message.read") == [{"chat": str(ALICE), "up_to": 12}] + await clock.advance(100) + assert runner.requests("message.read")[-1] == {"chat": str(ALICE), "up_to": 15} + + def test_reading_time_grows_with_the_text_and_is_capped(self): + from tlgr.actions.base import MessageFacts + + short = MessageFacts(chat_id=1, msg_id=1, text="hi") + long = MessageFacts(chat_id=1, msg_id=1, text="word " * 250) + huge = MessageFacts(chat_id=1, msg_id=1, text="word " * 5000) + media = MessageFacts(chat_id=1, msg_id=1, media="photo") + assert reading_seconds(short) < 1 + assert reading_seconds(long) == pytest.approx(60.0) + assert reading_seconds(huge) == 60.0 + assert reading_seconds(media) == 3.0 + + async def test_mentions_and_reactions_flags(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {"mentions": True, "reactions": True}}]) + await deliver(job, private_update(10, "")) + await clock.advance(1) + assert runner.requests("message.read")[0] == { + "chat": str(ALICE), + "up_to": 10, + "mentions": True, + "reactions": True, + } + + +class TestView: + async def test_a_channel_post_is_counted(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"view": {}}]) + await deliver(job, channel_update(40)) + await deliver(job, channel_update(41)) + await clock.advance(1) + assert runner.requests("message.view.get") == [ + {"chat": str(CHANNEL_ID), "msg_id": ["40", "41"], "increment": True} + ] + + async def test_a_voice_note_is_listened_to(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"view": True}]) + await deliver(job, private_update(10, "", media=voice_media())) + await deliver(job, private_update(11, "", media=voice_media(round_video=True))) + await clock.advance(1) + assert runner.requests("message.read") == [{"chat": str(ALICE), "contents": ["10", "11"]}] + + async def test_view_once_media_is_left_alone_unless_enabled(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"view": {}}]) + await deliver(job, private_update(10, "", media=voice_media(ttl=2147483647))) + await deliver(job, private_update(11, "just text")) + await clock.advance(1) + assert runner.calls == [] + assert _stats(scheduler)["skipped"] == 2 + + brave = _job(scheduler, [{"view": {"include_view_once": True}}]) + await deliver(brave, private_update(12, "", media=voice_media(ttl=2147483647))) + await clock.advance(1) + assert runner.requests("message.read") == [{"chat": str(ALICE), "contents": ["12"]}] + + +class TestReply: + async def test_typing_comes_first_and_lasts_as_long_as_the_text(self, sched): + scheduler, runner, clock = sched + text = "x" * 200 + job = _job(scheduler, [{"reply": text}]) + await deliver(job, private_update(10)) + await clock.advance(1) + assert runner.ops() == ["chat.typing"] + assert runner.requests("chat.typing")[0]["duration"] == typing_seconds(text) == 5.0 + await clock.advance(4.5) + assert runner.ops() == ["chat.typing", "message.send"] + assert runner.requests("message.send")[0] == { + "chat": str(ALICE), + "text": text, + "reply_to": 10, + "parse": "md", + } + + def test_typing_is_clamped(self): + assert typing_seconds("hi") == 2.0 + assert typing_seconds("x" * 10_000) == 15.0 + + async def test_typing_can_be_turned_off(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"reply": {"text": "hey", "typing": False}}]) + await deliver(job, private_update(10)) + await clock.advance(1) + assert runner.ops() == ["message.send"] + + async def test_no_typing_in_a_broadcast_channel(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"reply": "noted"}]) + await deliver(job, channel_update(40)) + await clock.advance(1) + assert runner.ops() == ["message.send"] + + +class TestPresence: + async def test_blip_goes_online_then_offline_after_the_action(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {}}], presence="blip") + await deliver(job, private_update(10)) + await clock.advance(1) + assert runner.ops() == ["profile.presence.set", "message.read"] + assert runner.requests("profile.presence.set")[0] == {"state": "online"} + await clock.advance(6) + assert runner.requests("profile.presence.set")[-1] == {"state": "offline"} + + async def test_overlapping_blips_merge(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {}}], presence="blip") + await deliver(job, private_update(10, user=ALICE)) + await deliver(job, private_update(11, user=ALICE + 1)) + await clock.run_for(15, step=0.5) + states = [r["state"] for r in runner.requests("profile.presence.set")] + assert states == ["online", "offline"] + + async def test_session_stays_online_while_more_is_due(self, sched): + scheduler, runner, clock = sched + job = _job( + scheduler, + [{"read": {}}, {"react": {"emoji": "👍", "delay": "8s"}}], + presence="session", + ) + await deliver(job, private_update(10)) + await clock.run_for(30, step=0.5) + states = [r["state"] for r in runner.requests("profile.presence.set")] + assert states == ["online", "offline"] + offline_at = runner.times("profile.presence.set")[-1] + assert offline_at > runner.times("reaction.add")[0] + + async def test_leave_never_touches_presence(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {}}]) + await deliver(job, private_update(10)) + await clock.advance(10) + assert "profile.presence.set" not in runner.ops() + + async def test_quiet_hours_hold_until_the_window_ends(self): + clock = FakeClock(start=datetime(2026, 1, 1, 2, 0).timestamp()) + scheduler, runner, clock = make_scheduler(clock=clock) + scheduler.start() + try: + job = _job( + scheduler, + [{"react": "👍"}, {"forward": {"to": "@archive"}}], + presence={"mode": "leave", "quiet_hours": "01:00-08:00"}, + on_takeover="ignore", + ) + await deliver(job, private_update(10)) + await clock.advance(5) + assert runner.ops() == ["message.forward"] + await clock.run_for(6 * 3600 - 60, step=600) + assert "reaction.add" not in runner.ops() + await clock.run_for(3600, step=30) + assert "reaction.add" in runner.ops() + released = runner.times("reaction.add")[0] + assert datetime.fromtimestamp(released).hour == 8 + finally: + await scheduler.stop(timeout=0.1) + + +class TestTakeover: + async def _queued(self, scheduler, on_takeover): + job = _job( + scheduler, + [ + {"read": {"delay": "60s"}}, + {"view": {"delay": "60s"}}, + {"react": {"emoji": "👍", "delay": "60s"}}, + {"reply": {"text": "hi", "delay": "60s"}}, + {"forward": {"to": "@archive", "delay": "60s"}}, + ], + on_takeover=on_takeover, + ) + await deliver(job, private_update(10, "", media=voice_media())) + return {item.action for item in scheduler.items.values()} + + async def test_cancel_drops_everything_but_the_forward(self, sched): + scheduler, runner, clock = sched + await self._queued(scheduler, "cancel") + update = types.UpdateReadHistoryInbox( + peer=types.PeerUser(user_id=ALICE), + max_id=10, + still_unread_count=0, + pts=2, + pts_count=1, + ) + await scheduler.on_bus(bus_envelope(), update) + assert {item.action for item in scheduler.items.values()} == {"forward"} + assert _stats(scheduler, 2)["superseded"] == 1 + + async def test_cancel_read_drops_only_read_and_view(self, sched): + scheduler, runner, clock = sched + await self._queued(scheduler, "cancel_read") + await scheduler.on_bus(bus_envelope(), private_update(30, "on it", out=True)) + assert {item.action for item in scheduler.items.values()} == { + "react", + "reply", + "forward", + } + + async def test_ignore_keeps_everything(self, sched): + scheduler, runner, clock = sched + before = await self._queued(scheduler, "ignore") + await scheduler.on_bus(bus_envelope(), private_update(30, "on it", out=True)) + assert {item.action for item in scheduler.items.values()} == before + + async def test_tlgrs_own_read_is_not_a_takeover(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {}}, {"reply": {"text": "hi", "delay": "60s"}}]) + await deliver(job, private_update(10)) + await clock.advance(1) + assert runner.ops() == ["message.read"] + echo = types.UpdateReadHistoryInbox( + peer=types.PeerUser(user_id=ALICE), + max_id=10, + still_unread_count=0, + pts=2, + pts_count=1, + ) + await scheduler.on_bus(bus_envelope(), echo) + assert [item.action for item in scheduler.items.values()] == ["reply"] + + async def test_the_read_that_follows_tlgrs_own_send_is_not_a_takeover(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {"delay": "60s"}}]) + await deliver(job, private_update(10)) + scheduler.begin_send(ALICE) + update = types.UpdateReadHistoryInbox( + peer=types.PeerUser(user_id=ALICE), + max_id=12, + still_unread_count=0, + pts=2, + pts_count=1, + ) + await scheduler.on_bus(bus_envelope(), update) + assert len(scheduler.items) == 1 + + async def test_a_read_in_one_forum_topic_leaves_other_topics_alone(self, sched): + from job_helpers import topic_update + + scheduler, runner, clock = sched + job = _job(scheduler, [{"reply": {"text": "hi", "delay": "60s"}}]) + await deliver(job, topic_update(70, topic=60)) + await deliver(job, topic_update(71, topic=50)) + update = types.UpdateReadChannelDiscussionInbox( + channel_id=5150, top_msg_id=60, read_max_id=80 + ) + await scheduler.on_bus(bus_envelope(), update) + assert [item.topic_id for item in scheduler.items.values()] == [50] + + async def test_a_message_after_the_read_point_survives(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"reply": {"text": "hi", "delay": "60s"}}]) + await deliver(job, private_update(10)) + await deliver(job, private_update(20)) + update = types.UpdateReadHistoryInbox( + peer=types.PeerUser(user_id=ALICE), + max_id=15, + still_unread_count=1, + pts=2, + pts_count=1, + ) + await scheduler.on_bus(bus_envelope(), update) + assert [item.msg_id for item in scheduler.items.values()] == [20] + + +class TestFailures: + async def test_flood_wait_reschedules_and_slows_the_queue(self, sched): + scheduler, runner, clock = sched + runner.fail("message.forward", errors.FloodWaitError(request=None, capture=30)) + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await clock.advance(1) + assert len(runner.requests("message.forward")) == 1 + await clock.advance(20) + assert len(runner.requests("message.forward")) == 1 + await clock.advance(20) + assert len(runner.requests("message.forward")) == 2 + assert _stats(scheduler)["done"] == 1 + assert scheduler.pacers["forward"].slowdown == 2.0 + + async def test_a_transient_failure_is_retried_with_backoff(self, sched): + scheduler, runner, clock = sched + runner.fail("message.forward", ConnectionError("reset"), ConnectionError("reset")) + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await clock.run_for(120, step=1) + assert len(runner.requests("message.forward")) == 3 + assert _stats(scheduler)["done"] == 1 + + async def test_an_expiring_action_gives_up_after_a_few_retries(self, sched): + scheduler, runner, clock = sched + runner.fail("reaction.add", *[ConnectionError("down")] * 10) + job = _job(scheduler, [{"react": "👍"}]) + await deliver(job, private_update(10)) + await clock.run_for(600, step=5) + assert len(runner.requests("reaction.add")) == 4 + assert _stats(scheduler)["errors"] == 1 + + async def test_a_forward_rides_out_a_long_reconnect(self, sched): + scheduler, runner, clock = sched + runner.fail("message.forward", *[ConnectionError("down")] * 6) + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await clock.run_for(3 * 3600, step=60) + assert len(runner.requests("message.forward")) == 7 + assert _stats(scheduler)["done"] == 1 + + async def test_a_request_cancelled_by_a_disconnect_is_retried(self, sched): + """Telethon cancels in-flight requests on disconnect; the worker must survive.""" + scheduler, runner, clock = sched + runner.fail("message.forward", asyncio.CancelledError()) + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await deliver(job, private_update(11, user=ALICE + 1)) + await clock.run_for(60, step=1) + assert len(runner.requests("message.forward")) == 3 + assert _stats(scheduler)["done"] == 2 + + async def test_a_flood_does_not_use_up_the_transient_retries(self, sched): + scheduler, runner, clock = sched + runner.fail( + "message.forward", + errors.FloodWaitError(request=None, capture=20), + ConnectionError("reset"), + ) + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await clock.run_for(60, step=1) + assert len(runner.requests("message.forward")) == 3 + item_stats = _stats(scheduler) + assert item_stats["done"] == 1 + + async def test_a_running_action_of_a_removed_job_is_not_requeued(self, sched): + scheduler, runner, clock = sched + gate = asyncio.Event() + + async def slow(request): + await gate.wait() + raise ConnectionError("reset") + + original = runner.__call__ + + async def call(op, request): + if op == "message.forward": + runner.calls.append((op, dict(request), clock.now())) + return await slow(request) + return await original(op, request) + + scheduler.runner = call + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await clock.advance(1) + assert scheduler.forget_job("dm") == 1 + gate.set() + await clock.run_for(60, step=1) + assert scheduler.items == {} + assert len(runner.requests("message.forward")) == 1 + + async def test_a_permanent_failure_is_counted(self, sched): + scheduler, runner, clock = sched + runner.fail("message.forward", errors.ChatWriteForbiddenError(request=None)) + job = _job(scheduler, [{"forward": {"to": "@archive"}}]) + await deliver(job, private_update(10)) + await clock.run_for(60, step=1) + assert len(runner.requests("message.forward")) == 1 + assert _stats(scheduler)["errors"] == 1 + + +class TestDryRun: + async def test_dry_run_schedules_counts_and_never_calls_telegram(self, sched, caplog): + caplog.set_level(logging.INFO, logger="tlgr.gateway") + scheduler, runner, clock = sched + job = _job(scheduler, [{"react": "👍"}, {"reply": "hi"}], dry_run=True) + await deliver(job, private_update(10)) + await clock.advance(5) + assert runner.calls == [] + assert _stats(scheduler, 0)["done"] == 1 + assert _stats(scheduler, 1)["done"] == 1 + assert "dry run: would read" in caplog.text + + +class TestCountersAndQueue: + async def test_counters_and_cancel(self, sched): + scheduler, runner, clock = sched + job = _job(scheduler, [{"read": {"delay": "60s"}}]) + await deliver(job, private_update(10, user=ALICE)) + await deliver(job, private_update(11, user=ALICE + 1)) + assert _stats(scheduler)["pending"] == 2 + cancelled = scheduler.cancel(chat_id=ALICE) + assert [item.msg_id for item in cancelled] == [10] + assert _stats(scheduler)["superseded"] == 1 + assert _stats(scheduler)["pending"] == 1 + assert scheduler.cancel() == [] + assert len(scheduler.cancel(everything=True)) == 1 + + async def test_stop_waits_for_nothing_when_idle(self, sched): + scheduler, runner, clock = sched + await asyncio.wait_for(scheduler.stop(timeout=0.1), timeout=2) diff --git a/tests/test_job_queue.py b/tests/test_job_queue.py new file mode 100644 index 0000000..5f716dc --- /dev/null +++ b/tests/test_job_queue.py @@ -0,0 +1,202 @@ +"""`job queue list/cancel`, the per-action counters in `job list`/`job get`, and +the validation `job add` and `job reload --validate-only` apply.""" + +from __future__ import annotations + +from typing import Any + +import pytest +from fake_telethon import make_user +from job_helpers import ALICE, BOB, daemon_job_env, private_update, until + +from tlgr.core.errors import EXIT_USAGE, classify + +JOBS = """\ +jobs: + - name: dm-ack + account: work + filters: {chat_type: private} + actions: + - read: {delay: 1h} + - react: {emoji: "👍", delay: 2h} + - name: archive + account: work + actions: + - forward: {to: ["@alice"], delay: 1h} +""" + + +async def call(client, in_thread, op: str, request: Any = None, **kwargs: Any) -> dict[str, Any]: + return await in_thread(client.op, op, request, **kwargs) + + +async def result(client, in_thread, op: str, request: Any = None, **kwargs: Any) -> Any: + return (await call(client, in_thread, op, request, **kwargs))["result"] + + +@pytest.fixture +async def queued(live_daemon, world, tlgr_home): + world.add_user(make_user(ALICE, username="alice")) + world.add_user(make_user(BOB, username="bobby")) + env = await daemon_job_env(live_daemon) + (tlgr_home / "jobs.yaml").write_text(JOBS) + await live_daemon.reload_jobs() + client = live_daemon.sessions.get("work").client + await client.feed(private_update(10, user=ALICE)) + await client.feed(private_update(11, user=BOB)) + await until(lambda: len(env.scheduler.items) == 6) + return env + + +class TestQueueList: + async def test_every_pending_action_is_listed_soonest_first(self, queued, client, in_thread): + rows = await result(client, in_thread, "job.queue.list") + assert len(rows) == 6 + assert [row["state"] for row in rows] == ["waiting"] * 6 + assert {row["action"] for row in rows} == {"read", "react", "forward"} + etas = [row["eta_s"] for row in rows] + assert etas == sorted(etas) + react = next(row for row in rows if row["action"] == "react") + assert react["detail"] == "👍" + assert react["due_at"].endswith("Z") + + async def test_filters(self, queued, client, in_thread): + rows = await result(client, in_thread, "job.queue.list", {"job": "archive"}) + assert {row["action"] for row in rows} == {"forward"} + rows = await result(client, in_thread, "job.queue.list", {"chat": "@bobby"}) + assert {row["chat_id"] for row in rows} == {BOB} + rows = await result(client, in_thread, "job.queue.list", {"chat": str(ALICE)}) + assert len(rows) == 3 + + +class TestQueueCancel: + async def test_by_id(self, queued, client, in_thread): + rows = await result(client, in_thread, "job.queue.list") + body = await result(client, in_thread, "job.queue.cancel", {"ids": [rows[0]["id"]]}) + assert body == {"cancelled": 1, "ids": [rows[0]["id"]]} + assert len(queued.scheduler.items) == 5 + + async def test_by_chat_and_job(self, queued, client, in_thread): + body = await result( + client, in_thread, "job.queue.cancel", {"chat": "@alice", "job": "dm-ack"} + ) + assert body["cancelled"] == 2 + remaining = {(item.job, item.chat_id) for item in queued.scheduler.items.values()} + assert ("dm-ack", ALICE) not in remaining + + async def test_all(self, queued, client, in_thread): + body = await result(client, in_thread, "job.queue.cancel", {"every": True}) + assert body["cancelled"] == 6 + assert queued.scheduler.items == {} + + async def test_a_selector_is_required(self, queued, client, in_thread): + with pytest.raises(Exception) as caught: + await call(client, in_thread, "job.queue.cancel", {}) + assert classify(caught.value).exit_code == EXIT_USAGE + + async def test_the_cli_wants_yes_off_a_terminal(self, queued, in_thread): + from click.testing import CliRunner + + from tlgr.cli import cli + + outcome = await in_thread(CliRunner().invoke, cli, ["job", "queue", "cancel", "--all"]) + assert outcome.exit_code == EXIT_USAGE + assert len(queued.scheduler.items) == 6 + outcome = await in_thread( + CliRunner().invoke, cli, ["job", "queue", "cancel", "--all", "--yes"] + ) + assert outcome.exit_code == 0, outcome.output + assert queued.scheduler.items == {} + + async def test_bare_job_queue_lists(self, queued, in_thread): + from click.testing import CliRunner + + from tlgr.cli import cli + + outcome = await in_thread(CliRunner().invoke, cli, ["job", "queue"]) + assert outcome.exit_code == 0, outcome.output + assert "dm-ack" in outcome.output + assert "archive" in outcome.output + + +class TestCounters: + async def test_job_list_and_get_report_per_action_counters(self, queued, client, in_thread): + queued.scheduler.cancel(chat_id=BOB, job="dm-ack") + rows = await result(client, in_thread, "job.list") + ack = next(row for row in rows if row["name"] == "dm-ack") + assert ack["running"] is True + assert ack["matched"] == 2 + assert ack["pending"] == 2 + assert ack["superseded"] == 2 + by_action = {c["action"]: c for c in ack["action_counters"]} + assert by_action["read"]["pending"] == 1 + assert by_action["react"]["superseded"] == 1 + for key in ("done", "skipped", "expired", "errors"): + assert by_action["read"][key] == 0 + + state = await result(client, in_thread, "job.get", {"name": "dm-ack"}) + assert state["pending"] == 2 + assert [c["action"] for c in state["action_counters"]] == ["read", "react"] + + async def test_daemon_status_carries_the_queue(self, queued, client, in_thread): + status = await in_thread(client.status) + assert status["actions"]["work"]["pending"] == 6 + assert status["actions"]["work"]["by_action"]["react"] == 2 + + +class TestJobLifecycle: + async def test_disable_drops_what_the_job_queued(self, queued, client, in_thread): + await call(client, in_thread, "job.disable", {"name": "archive"}) + assert {item.job for item in queued.scheduler.items.values()} == {"dm-ack"} + stats = queued.scheduler.job_stats("archive") + assert stats[0]["superseded"] == 2 + + async def test_remove_drops_the_queue_and_the_counters(self, queued, client, in_thread): + await call(client, in_thread, "job.remove", {"name": "dm-ack"}) + assert {item.job for item in queued.scheduler.items.values()} == {"archive"} + assert queued.scheduler.job_stats("dm-ack") == [] + + +class TestValidation: + async def test_job_add_refuses_bad_knobs(self, live_daemon, client, in_thread): + bad = { + "name": "dm", + "action": ["react:emoji=👍,percent=150"], + } + with pytest.raises(Exception) as caught: + await call(client, in_thread, "job.add", bad) + error = classify(caught.value) + assert error.exit_code == EXIT_USAGE + assert "percent" in error.message + + async def test_job_add_accepts_the_new_actions(self, live_daemon, client, in_thread, tlgr_home): + body = { + "name": "dm", + "action": ["read:delay=10-90s", "react:emoji=👍", "view"], + "knob": ["presence=session", "on_takeover=cancel_read"], + } + await call(client, in_thread, "job.add", body) + import yaml + + saved = yaml.safe_load((tlgr_home / "jobs.yaml").read_text())["jobs"][0] + assert saved["actions"] == [ + {"read": {"delay": "10-90s"}}, + {"react": {"emoji": "👍"}}, + {"view": {}}, + ] + assert saved["presence"] == "session" + + async def test_reload_validate_only_reports_every_problem( + self, live_daemon, client, in_thread, tlgr_home + ): + (tlgr_home / "jobs.yaml").write_text( + "pacing:\n work:\n react: {every: soon}\n" + "jobs:\n" + " - name: a\n actions: [{react: {emoji: '👍', delay: 'forever'}}]\n" + " - name: b\n presence: loud\n actions: [{read: {}}]\n" + ) + body = await result(client, in_thread, "job.reload", {"validate_only": True}) + errors = " | ".join(body["errors"]) + assert "pacing.work.react.every" in errors + assert "job 'a', action 1: delay" in errors + assert "job 'b': presence: unknown mode 'loud'" in errors diff --git a/tests/test_ops_daemon.py b/tests/test_ops_daemon.py index f801351..381638d 100644 --- a/tests/test_ops_daemon.py +++ b/tests/test_ops_daemon.py @@ -476,6 +476,14 @@ def test_an_unknown_event_name_in_the_webhook_is_an_error(self, tlgr_home): assert report.ok is False assert "unknown event type" in report.errors[0].message + def test_a_bad_job_knob_is_an_error(self, tlgr_home): + (tlgr_home / "jobs.yaml").write_text( + "jobs:\n - name: dm\n actions:\n - react: {emoji: '👍', percent: 140}\n" + ) + report = local("config.validate", {"file": "jobs"}) + assert report.ok is False + assert "percent" in report.errors[0].message + class TestConfigServer: async def test_it_reads_the_server_limits(self, live_daemon, client, in_thread): diff --git a/tlgr/actions/README.md b/tlgr/actions/README.md index 9cbb0a3..56ae30a 100644 --- a/tlgr/actions/README.md +++ b/tlgr/actions/README.md @@ -1,117 +1,170 @@ # Actions -Registry-based pipeline actions for the Gateway. Actions are the final stage -of the pipeline -- they execute side effects (sending messages, forwarding, -etc.) based on the event that passed through filters and processors. +The last stage of a gateway job: what it does with a message that passed its +filters. Every built-in action runs through the op layer (the same +operations the CLI calls), in process, for the job's account, so a job gets +the policy allow/deny list, the rate limiter, the flood-wait budget and the +self-origin events on the bus, exactly like a command typed by hand. ## How it works ```mermaid flowchart TB - E["Event passes filters"] --> LOOP["For each action in the job's action list"] - LOOP --> CHECK["Check per-action filters
(optional)"] - CHECK --> RESOLVE["Resolve processor chain
(per-action overrides job-level)"] - RESOLVE --> EXEC["Execute
action_func(event, config, client, chain)"] + E["Event passes the job's filters"] --> LOOP["For each action"] + LOOP --> CHECK["Per-action filters"] + CHECK --> ROLL["percent roll"] + ROLL --> PLAN["plan(): payloads, on the bus lane, no awaits"] + PLAN --> Q["Account scheduler: delay, quiet hours, pacer, persistence"] + Q --> EXEC["execute(): operations through dispatch, when due"] ``` -Every action is an async function registered with `@register_action`. Actions -receive: -- `event` -- the `Event` envelope -- `config` -- the action's YAML config value -- `client` -- a `ClientWrapper` for Telegram API calls -- `chain` -- an optional `ProcessorChain` for text modification +An action is an `Action` subclass (`base.py`) registered with +`@register_action`. It is split in two because the halves happen at +different times, possibly in different processes: + +* `plan(facts, params, chain, rng)` runs when the event arrives. It reads + the message through `MessageFacts`, applies processors, picks the emoji, + and returns plain payloads. It never awaits, so the bus lane is never held. +* `execute(items, runtime)` runs when the item is due and its pacer slot is + open, possibly after a restart, from nothing but the persisted + `PendingItem`. It calls operations through `runtime.op(...)`. + +Class attributes tell the scheduler how to treat the kind: its expiry, whether +quiet hours hold it, which `on_takeover` modes cancel it, whether an album +gets one action, and how items coalesce (`batch_key`). + +A plain async function registered with `@register_action` is still accepted. +It receives `(event, config, client, chain)` and runs at once, outside the +scheduler, as actions always did. ## Built-in actions -### reply +Every action also takes the shared knobs `delay`, `percent`, `presence`, +`on_takeover`, `dry_run` and `filters`; see `tlgr/gateway/README.md`. -Sends a static text reply to the triggering message. +### forward ```yaml -actions: - - reply: "shut up i'm just a bot!" +- forward: "@archive" +- forward: + to: ["@clean_feed", "@archive"] + drop_author: true + processors: [strip_formatting] ``` -If a processor chain is provided, it's applied to the reply text before sending. +| Key | Type | Description | +|-----|------|-------------| +| `to` | `str` or `list` | Destination chat(s); each is its own pending item | +| `drop_author` | `bool` | Hide the original author (native forward) | +| `processors` | `list` | Rewrite the text; turns the forward into a re-send | -### forward +Without processors: `message.forward` (`--no-author` with `drop_author`). +With processors the processed text is re-sent with `message.send`, or a photo +or document is re-sent with the processed caption (`media.upload +--from-message`); text keeps its formatting as markdown. A link preview is +regenerated from the text. Service messages, empty messages and +self-destructing media are skipped. Pacing: 1 per 1.5 s. Never expires, never +held by quiet hours, never cancelled by a takeover. -Forwards the message to one or more destinations. +### reply ```yaml -actions: - - forward: - to: ["@clean_feed", "@archive"] - drop_author: true - processors: - - strip_formatting +- reply: "Away until Monday" +- reply: {text: "Got it", typing: true, delay: 1-3m, processors: [...]} ``` -| Config key | Type | Description | -|------------|------|-------------| -| `to` | `str` or `list[str]` | Destination chat(s) | -| `drop_author` | `bool` | Remove original author attribution | -| `processors` | `list` | Per-action processors (override job-level) | -| `filters` | `dict` | Per-action filters (AND'd with job-level) | - -When processors are present, the message text is transformed and sent as a new -message (or caption for media). Without processors, the message is forwarded -natively via Telegram's forward API. +| Key | Type | Description | +|-----|------|-------------| +| `text` | `str` | The reply; processors apply to it | +| `typing` | `bool` | Show "typing..." first (default `true`) | -## Per-action overrides +Typing lasts about 40 characters a second of the text, clamped to 2-15 s +(`chat.typing`), then `message.send` goes out as a reply to the message. The +reply queue is not held while typing: the item is re-queued for the moment +typing ends. No indicator in a broadcast channel. An album gets one reply. +Pacing: 1 per 1.5 s. Expires after 24 h. -Each action can have its own `filters` and `processors` that specialize the -job-level pipeline: +### react ```yaml -jobs: - - name: selective - filters: - chat_type: private - actions: - # This action only runs for messages with media - - forward: - to: ["@media_archive"] - filters: - has_media: true - - # This action runs for all private messages - - reply: "got your message!" +- react: "👍" +- react: ["👍", "❤", "🔥"] # one, uniformly at random, per message +- react: {"👍": 3, "🔥": 1} # weighted +- react: {emoji: ["👍", "🔥"], big: false, percent: 60, delay: 30-300s} +- react: "custom:5368324170671202286" # a custom (Premium) emoji ``` -Per-action filters are AND'd with the job-level filters. Per-action processors -replace (not extend) the job-level processor chain. +The job's choice replaces any reaction the account already had on the +message (`reaction.add --replace`). A chat that does not allow the emoji is +not checked first: the reaction is sent and REACTION_INVALID shows in the +action's `errors`. Knobs need the long form (a weighted mapping is all +emoji). -## Adding a custom action +Reacting implies reading: before the reaction goes out the chat is read up +to that message, coalesced with any read already pending for the chat, so +the other side never sees a reaction on a message still shown unread. This +happens even if the job has no `read` action. An album gets one reaction, on +the message with the caption, else the first one (where Telegram Desktop and +Android attach album reactions). Pacing: 1 per 4 s, at most 300 an hour. +Expires after 24 h. -1. Create an async function with signature - `(event: Event, config: Any, client: ClientWrapper, chain: ProcessorChain | None)`. -2. Decorate it with `@register_action("name")`. -3. Import it in `__init__.py`. +### read -```python -# tlgr/actions/react.py -from tlgr.actions import register_action +```yaml +- read: {} +- read: true +- read: {delay: 10-90s, mentions: true, reactions: true} +``` + +Marks the chat read up to the triggering message, never further (never +`max_id=0`). `message.read` picks the RPC for the peer: `readHistory` for a +private chat or basic group, `channels.readHistory` for a channel or +supergroup, `readDiscussion` for a forum topic. Reads that are due together +for one chat go out as one call with the highest id. On top of `delay`, the +item waits about as long as reading the text takes (250 words a minute, plus +3 s for media, at most 60 s). `mentions` and `reactions` also clear those +badges. Pacing: 1 per 2 s. Never expires. +### view -@register_action("react") -async def action_react(event, config, client, chain=None): - if event.source != "telegram": - return - emoji = str(config) if isinstance(config, str) else config.get("emoji", "👍") - msg = event.raw.message - await client.react_to_message(event.raw.chat_id, msg.id, emoji) +```yaml +- view: {} +- view: {include_view_once: true} ``` -Then import in `__init__.py`: +In a broadcast channel: `message.view.get --increment` (getMessagesViews with +`increment=true`), one call per chat with every id that is due. In a private +chat or group: voice notes and round video notes are marked listened +(`message.read --contents`). View-once and self-destructing media is never +consumed unless `include_view_once: true`. A message with nothing to view +counts as `skipped`. `view` and `read` are independent. Pacing: 1 per 2 s. +Expires after 24 h. + +## Writing an action ```python -from tlgr.actions import react # noqa: F401 -``` +from tlgr.actions import register_action +from tlgr.actions.base import Action, Outcome -Now usable in YAML: -```yaml -actions: - - react: "👍" +@register_action("pin") +class Pin(Action): + name = "pin" + cancelled_by = frozenset({"cancel"}) + + def parse(self, config): + return {"notify": bool((config or {}).get("notify", False))} + + def plan(self, facts, params, chain, rng): + return [{"notify": params["notify"]}] + + async def execute(self, items, rt): + item = items[0] + request = {"chat": str(item.chat_id), "msg_id": item.msg_id, **item.payload} + await rt.op("message.pin", request) + return Outcome() ``` + +Add a pacer rule for it to `DEFAULT_PACING` in `gateway/pacer.py` (the +scheduler runs one queue per entry there) and it is paced, persisted and +counted like the built-ins. diff --git a/tlgr/actions/__init__.py b/tlgr/actions/__init__.py index c1f3994..9a05ece 100644 --- a/tlgr/actions/__init__.py +++ b/tlgr/actions/__init__.py @@ -1,9 +1,16 @@ """Registry-based actions for the Gateway pipeline. -Every action is an async function registered via ``@register_action``. -Actions receive an :class:`~tlgr.gateway.event.Event`, the action's config -from YAML, a :class:`~tlgr.jobs.client.JobClient`, and an optional -:class:`~tlgr.processors.ProcessorChain`. +A built-in action is an `Action` subclass registered with +``@register_action``: it plans payloads when the event arrives and executes +them through the op layer when the scheduler says they are due (see +`actions/base.py` and `gateway/scheduler.py`). Every built-in action is +therefore paced, persisted across restarts, and subject to the same policy, +rate limits and flood budgets as the CLI. + +A plain async function registered the same way is still accepted, for +actions written against the old interface: it receives the +:class:`~tlgr.gateway.event.Event`, the action's config, the job client and +the processor chain, and runs at once, outside the scheduler. """ from __future__ import annotations @@ -11,6 +18,7 @@ from collections.abc import Awaitable, Callable from typing import Any +from tlgr.actions.base import Action from tlgr.gateway.event import Event from tlgr.jobs.client import JobClient from tlgr.processors import ProcessorChain @@ -20,26 +28,37 @@ Awaitable[None], ] -_REGISTRY: dict[str, ActionFunc] = {} +_REGISTRY: dict[str, Action | ActionFunc] = {} -def register_action(name: str): - """Decorator that registers an action function under *name*.""" +def register_action(name: str) -> Callable[[Any], Any]: + """Register an `Action` subclass (instantiated once) or a legacy function.""" - def decorator(func: ActionFunc) -> ActionFunc: - _REGISTRY[name] = func - return func + def decorator(obj: Any) -> Any: + if isinstance(obj, type) and issubclass(obj, Action): + instance = obj() + instance.name = instance.name or name + _REGISTRY[name] = instance + else: + _REGISTRY[name] = obj + return obj return decorator -def get_action(name: str) -> ActionFunc | None: +def get_action(name: str) -> Action | ActionFunc | None: return _REGISTRY.get(name) +def get_builtin(name: str) -> Action | None: + """The scheduled `Action` registered under *name*, or None.""" + found = _REGISTRY.get(name) + return found if isinstance(found, Action) else None + + def list_actions() -> list[str]: return list(_REGISTRY.keys()) # Import built-in action modules so they self-register. -from tlgr.actions import forward, reply # noqa: E402, F401 +from tlgr.actions import forward, react, read, reply, view # noqa: E402, F401 diff --git a/tlgr/actions/base.py b/tlgr/actions/base.py new file mode 100644 index 0000000..a66f8d6 --- /dev/null +++ b/tlgr/actions/base.py @@ -0,0 +1,225 @@ +"""What a built-in job action is: plan at event time, execute when due. + +An action is split in two because the two halves happen at different times, +possibly in different processes: + +* `plan` runs on the bus lane when the event arrives. It reads the Telethon + event (through `MessageFacts`), applies processors, picks the emoji, and + returns plain payloads. It must not await anything slow; it does not await + at all. +* `execute` runs when the item is due and its pacer slot is open, maybe after + a restart, from nothing but the persisted `PendingItem`. It calls + operations through the runtime (`rt.op("reaction.add", ...)`), never + Telethon. + +The class attributes tell the scheduler how to treat the action: which pacer +queue it uses, when it expires, whether quiet hours hold it, which takeover +modes cancel it, and how its items coalesce. +""" + +from __future__ import annotations + +import random +from dataclasses import dataclass +from typing import Any, Protocol + +from tlgr.gateway.knobs import KnobError +from tlgr.processors import ProcessorChain + +__all__ = ["Action", "ActionError", "MessageFacts", "Outcome", "Runtime", "facts_from_event"] + +HOUR = 3600.0 + + +class ActionError(KnobError): + """An action config that cannot work. The message names the key.""" + + +@dataclass +class Outcome: + """What happened to a batch: `done`, `skipped`, or `later` (re-queued at `due_at`).""" + + status: str = "done" + due_at: float | None = None + + +@dataclass +class MessageFacts: + """What the actions need from a message, read once from the Telethon event.""" + + chat_id: int + msg_id: int + peer: str = "user" + topic_id: int | None = None + grouped_id: int | None = None + #: `message.text`: markdown when the client has a parse mode, so a + #: processed re-send keeps its formatting the way the old action did. + text: str = "" + #: `photo`, `document`, `webpage`, `other`, or None. + media: str | None = None + voice_or_round: bool = False + view_once: bool = False + service: bool = False + + +class Runtime(Protocol): + """The scheduler services an executing action may use.""" + + rng: random.Random + + def now(self) -> float: ... + + async def op(self, op: str, request: dict[str, Any]) -> Any: ... + + async def ensure_read(self, item: Any) -> None: ... + + def read_mark(self, chat_id: int, topic_id: int | None) -> int: ... + + def begin_read(self, chat_id: int, topic_id: int | None, max_id: int) -> None: ... + + def note_read(self, chat_id: int, topic_id: int | None, max_id: int) -> None: ... + + def begin_send(self, chat_id: int) -> None: ... + + def note_sent(self, chat_id: int, ids: list[int]) -> None: ... + + def spawn(self, coro: Any, *, what: str) -> None: ... + + +class Action: + """One built-in action kind. Subclasses set the attributes and the hooks.""" + + name: str = "" + #: Seconds after the event arrived when a pending item stops being worth + #: doing; None means never. Overridable per account in `pacing.expire`. + default_expiry: float | None = None + #: Quiet hours hold this action until the window ends. + quiet_hold: bool = True + #: The `on_takeover` modes that cancel a pending item of this kind. + cancelled_by: frozenset[str] = frozenset() + #: One action per album (the official apps attach album reactions to one + #: message), instead of one per message of the group. + album_once: bool = False + accepts_processors: bool = False + + def parse(self, config: Any) -> dict[str, Any]: + """Validate the action's own config (knobs already removed).""" + return {} + + def plan( + self, + facts: MessageFacts, + params: dict[str, Any], + chain: ProcessorChain | None, + rng: random.Random, + ) -> list[dict[str, Any]]: + """Payloads to schedule; an empty list means there is nothing to do.""" + return [{}] + + def extra_delay(self, facts: MessageFacts, params: dict[str, Any]) -> float: + """Seconds added on top of the `delay` knob.""" + return 0.0 + + def key_extra(self, payload: dict[str, Any]) -> str: + """What distinguishes two payloads planned from one message.""" + return "" + + def batch_key(self, item: Any) -> tuple[Any, ...] | None: + """Items with the same key that are due together run as one call.""" + return None + + async def execute(self, items: list[Any], rt: Runtime) -> Outcome: + raise NotImplementedError + + def describe(self, items: list[Any]) -> str: + first = items[0] + return f"{self.name} chat {first.chat_id} msg {first.msg_id}" + + +def _peer_kind(tg_event: Any, message: Any) -> str: + peer = getattr(message, "peer_id", None) + kind = type(peer).__name__ + if kind == "PeerUser": + return "user" + if kind == "PeerChat": + return "chat" + if getattr(message, "post", False): + return "channel" + chat = getattr(tg_event, "chat", None) + if getattr(chat, "broadcast", False): + return "channel" + return "megagroup" + + +def _topic_of(message: Any) -> int | None: + reply = getattr(message, "reply_to", None) + if reply is None or not getattr(reply, "forum_topic", False): + return None + top = getattr(reply, "reply_to_top_id", None) or getattr(reply, "reply_to_msg_id", None) + return int(top) if top else None + + +def _media_kind(media: Any) -> str | None: + if media is None: + return None + name = type(media).__name__ + if name == "MessageMediaPhoto": + return "photo" + if name == "MessageMediaDocument": + return "document" + if name == "MessageMediaWebPage": + return "webpage" + if name in ("MessageMediaEmpty", "MessageMediaUnsupported"): + return None + return "other" + + +def _voice_or_round(media: Any) -> bool: + if type(media).__name__ != "MessageMediaDocument": + return False + if getattr(media, "voice", False) or getattr(media, "round", False): + return True + document = getattr(media, "document", None) + for attribute in getattr(document, "attributes", None) or []: + name = type(attribute).__name__ + if name == "DocumentAttributeAudio" and getattr(attribute, "voice", False): + return True + if name == "DocumentAttributeVideo" and getattr(attribute, "round_message", False): + return True + return False + + +def facts_from_event(tg_event: Any) -> MessageFacts: + message = tg_event.message + media = getattr(message, "media", None) + text = getattr(message, "text", None) + if not isinstance(text, str): + text = getattr(message, "message", "") or "" + return MessageFacts( + chat_id=int(tg_event.chat_id), + msg_id=int(message.id), + peer=_peer_kind(tg_event, message), + topic_id=_topic_of(message), + grouped_id=getattr(message, "grouped_id", None), + text=text or "", + media=_media_kind(media), + voice_or_round=_voice_or_round(media), + view_once=bool(getattr(media, "ttl_seconds", None)), + service=getattr(message, "action", None) is not None, + ) + + +def check_keys(name: str, config: dict[str, Any], allowed: set[str]) -> None: + unknown = sorted(set(config) - allowed) + if unknown: + own = ", ".join(sorted(allowed)) or "none" + raise ActionError( + f"{name}: unknown key(s) {unknown}. Its own keys: {own}; every action also " + "takes filters, delay, percent, presence, on_takeover and dry_run" + ) + + +def require_bool(name: str, key: str, value: Any) -> bool: + if not isinstance(value, bool): + raise ActionError(f"{name}: {key} must be true or false, got {value!r}") + return value diff --git a/tlgr/actions/forward.py b/tlgr/actions/forward.py index ecf647c..d34f156 100644 --- a/tlgr/actions/forward.py +++ b/tlgr/actions/forward.py @@ -1,74 +1,123 @@ -"""Forward action — relay messages to one or more destinations.""" +"""Forward action: relay messages to one or more destinations. + +Without processors it is a native forward (`message.forward`, with +`--no-author` when `drop_author` is set). With processors the text is +rewritten, so a forward would carry the original; instead the processed text +is re-sent (`message.send`), or, for a photo or document, the same media is +re-sent with the processed caption (`media.upload --from-message`). Text is +taken and sent as markdown, the client's parse mode, so formatting survives +the rewrite as it always has. + +Each destination is its own pending item, so one destination refusing the +message (CHAT_WRITE_FORBIDDEN, a private channel) does not stop the others, +and each is retried or counted on its own. + +Forwards never expire, are never held by quiet hours and are never cancelled +by a manual takeover: a relay that silently drops posts is broken. +""" from __future__ import annotations -import asyncio -import logging +import random from typing import Any -from telethon import errors - from tlgr.actions import register_action -from tlgr.filters.message import is_forwardable -from tlgr.gateway.event import Event -from tlgr.jobs.client import JobClient +from tlgr.actions.base import ( + Action, + ActionError, + MessageFacts, + Outcome, + Runtime, + check_keys, + require_bool, +) from tlgr.processors import ProcessorChain -log = logging.getLogger("tlgr.actions.forward") - @register_action("forward") -async def action_forward( - event: Event, - config: Any, - client: JobClient, - chain: ProcessorChain | None = None, -) -> None: - if event.source != "telegram": - log.warning("forward action only supports telegram events") - return +class Forward(Action): + name = "forward" + quiet_hold = False + accepts_processors = True - message = event.raw.message - - ok, reason = is_forwardable(message) - if not ok: - log.debug("message not forwardable: %s", reason) - return - - if isinstance(config, str): - destinations = [config] + def parse(self, config: Any) -> dict[str, Any]: drop_author = False - elif isinstance(config, dict): - to = config.get("to", []) - destinations = to if isinstance(to, list) else [to] - drop_author = config.get("drop_author", False) - else: - log.warning("invalid forward config: %r", config) - return - - for i, dest_ref in enumerate(destinations): - try: - dest_id = await client.resolve_chat(dest_ref) + if isinstance(config, dict): + check_keys(self.name, config, {"to", "drop_author"}) + to = config.get("to") + if "drop_author" in config: + drop_author = require_bool(self.name, "drop_author", config["drop_author"]) + else: + to = config + destinations = to if isinstance(to, list) else [to] if to not in (None, "") else [] + if not destinations or any( + isinstance(d, bool) or not isinstance(d, (str, int)) or d == "" for d in destinations + ): + raise ActionError(f"{self.name}: `to` must name one or more chats, got {to!r}") + return {"to": [str(d) for d in destinations], "drop_author": drop_author} - if chain: - original = message.text or getattr(message, "message", "") or "" - transformed = chain.apply(original) if original else "" - if message.media: - await client.client.send_file(dest_id, message.media, caption=transformed) - else: - await client.client.send_message(dest_id, transformed) + def plan( + self, + facts: MessageFacts, + params: dict[str, Any], + chain: ProcessorChain | None, + rng: random.Random, + ) -> list[dict[str, Any]]: + # What `is_forwardable` refused before: service messages, empty ones, + # and self-destructing media. + if facts.service or facts.view_once or (not facts.text and facts.media is None): + return [] + resend: dict[str, Any] | None = None + if chain: + text = chain.apply(facts.text) if facts.text else "" + if facts.media in ("photo", "document"): + resend = {"text": text, "media": True} + elif text: + # A link preview is regenerated from the text; it is not media + # that can be re-sent. + resend = {"text": text, "media": False} else: - await client.client.forward_messages( - dest_id, - message, - drop_author=drop_author, - ) - except errors.ChatWriteForbiddenError: - log.warning("cannot write to %s", dest_ref) - except errors.ChannelPrivateError: - log.warning("channel %s is private", dest_ref) - except Exception as e: - log.error("forward to %s failed: %s", dest_ref, e) + return [] + return [ + {"to": destination, "drop_author": params["drop_author"], "resend": resend} + for destination in params["to"] + ] + + def key_extra(self, payload: dict[str, Any]) -> str: + return str(payload.get("to", "")) + + async def execute(self, items: list[Any], rt: Runtime) -> Outcome: + item = items[0] + payload = item.payload + resend = payload.get("resend") + if not resend: + result = await rt.op( + "message.forward", + { + "chat": str(item.chat_id), + "msg_id": [str(item.msg_id)], + "to": [payload["to"]], + "no_author": bool(payload.get("drop_author")), + }, + ) + for row in (result or {}).get("items") or []: + rt.note_sent(int(row.get("chat_id") or 0), [int(row.get("id") or 0)]) + elif resend.get("media"): + request: dict[str, Any] = { + "chat": payload["to"], + "from_message": f"{item.chat_id}:{item.msg_id}", + "parse": "md", + } + if resend.get("text"): + request["caption"] = [resend["text"]] + await rt.op("media.upload", request) + else: + await rt.op( + "message.send", {"chat": payload["to"], "text": resend["text"], "parse": "md"} + ) + return Outcome() - if i < len(destinations) - 1: - await asyncio.sleep(0.3) + def describe(self, items: list[Any]) -> str: + item = items[0] + how = "re-send" if item.payload.get("resend") else "forward" + return f"{how} msg {item.msg_id} from {item.chat_id} to {item.payload.get('to')}" diff --git a/tlgr/actions/react.py b/tlgr/actions/react.py new file mode 100644 index 0000000..59aa798 --- /dev/null +++ b/tlgr/actions/react.py @@ -0,0 +1,133 @@ +"""React action: put a reaction on the triggering message. + +`react: "👍"`, `react: ["👍", "❤", "🔥"]` (one picked uniformly per message), +`react: {"👍": 3, "🔥": 1}` (weighted), or the long form +`react: {emoji: ..., big: false, percent: 60, delay: 30-300s}`. A custom +(Premium) emoji is written `custom:`, the spelling `reaction +add` uses. + +The job's choice replaces whatever reaction the account already had +(`reaction.add --replace`). A chat that does not allow the emoji is not +checked first: the reaction is sent and REACTION_INVALID lands in the +action's error counter, which is where a misconfigured job should show up. + +Reacting implies reading. Before the reaction goes out the chat is read up +to that message (coalesced with any read already pending for the chat), so +the other side never sees a reaction on a message their client still shows +as unread. That happens whether or not the job has a `read` action. + +An album (one `grouped_id`, delivered as several messages) gets one reaction, +on the message carrying the caption, else the first one. That is where +Telegram Desktop attaches an album's reactions (`GroupedMedia::itemForText`) +and where Telegram for Android looks (`findPrimaryMessageObject`). +""" + +from __future__ import annotations + +import random +from typing import Any + +from tlgr.actions import register_action +from tlgr.actions.base import ( + HOUR, + Action, + ActionError, + MessageFacts, + Outcome, + Runtime, + check_keys, + require_bool, +) +from tlgr.processors import ProcessorChain + +_CUSTOM = "custom:" + + +def _emoji(value: Any) -> str: + if not isinstance(value, str) or not value.strip(): + raise ActionError(f"react: {value!r} is not an emoji") + value = value.strip() + if value.startswith(_CUSTOM) and not value[len(_CUSTOM) :].isdigit(): + raise ActionError(f"react: {value!r} is not custom:") + return str(value) + + +def _choices(value: Any) -> list[list[Any]]: + """`[[emoji, weight], ...]` from a string, a list or a weighted mapping.""" + if isinstance(value, str): + return [[_emoji(value), 1.0]] + if isinstance(value, list): + if not value: + raise ActionError("react: the emoji list is empty") + return [[_emoji(item), 1.0] for item in value] + if isinstance(value, dict): + if not value: + raise ActionError("react: the emoji mapping is empty") + out: list[list[Any]] = [] + for emoji, weight in value.items(): + if isinstance(weight, bool) or not isinstance(weight, (int, float)) or weight <= 0: + raise ActionError( + f"react: the weight of {emoji!r} must be a positive number, got {weight!r}. " + "Knobs such as delay or percent go in the long form: " + "react: {emoji: {...}, delay: ...}" + ) + out.append([_emoji(emoji), float(weight)]) + return out + raise ActionError(f"react: {value!r} is not an emoji, a list or a weighted mapping") + + +@register_action("react") +class React(Action): + name = "react" + default_expiry = 24 * HOUR + cancelled_by = frozenset({"cancel"}) + album_once = True + + def parse(self, config: Any) -> dict[str, Any]: + big = False + if isinstance(config, dict) and "emoji" in config: + check_keys(self.name, config, {"emoji", "big"}) + if "big" in config: + big = require_bool(self.name, "big", config["big"]) + choices = _choices(config["emoji"]) + else: + choices = _choices(config) + return {"choices": choices, "big": big} + + def plan( + self, + facts: MessageFacts, + params: dict[str, Any], + chain: ProcessorChain | None, + rng: random.Random, + ) -> list[dict[str, Any]]: + if facts.service: + return [] + choices = params["choices"] + if len(choices) == 1: + emoji = choices[0][0] + else: + emoji = rng.choices([c[0] for c in choices], weights=[c[1] for c in choices])[0] + return [{"emoji": emoji, "big": params["big"]}] + + async def execute(self, items: list[Any], rt: Runtime) -> Outcome: + item = items[0] + await rt.ensure_read(item) + await rt.op( + "reaction.add", + { + "chat": str(item.chat_id), + "msg_id": item.msg_id, + "emoji": [item.payload["emoji"]], + "replace": True, + "big": bool(item.payload.get("big")), + }, + ) + return Outcome() + + def describe(self, items: list[Any]) -> str: + item = items[0] + return ( + f"read {item.chat_id} up to {item.msg_id}, " + f"then react {item.payload.get('emoji')} to msg {item.msg_id}" + ) diff --git a/tlgr/actions/read.py b/tlgr/actions/read.py new file mode 100644 index 0000000..4b895cf --- /dev/null +++ b/tlgr/actions/read.py @@ -0,0 +1,102 @@ +"""Read action: mark the chat read up to the triggering message. + +Never `max_id=0` ("everything"), and never past the triggering message: a +message that arrived after it may not have been "seen" yet. `message.read` +picks the RPC for the peer (readHistory for a private chat or basic group, +channels.readHistory for a channel or supergroup, readDiscussion for a forum +topic), so a channel read is never the silent no-op `messages.readHistory` +is there. + +Reads coalesce per chat: every read that has come due for a chat goes out +as one call with the highest id, each item still keeping its own delay. A +read never expires; it is a watermark, so doing it late is harmless. + +On top of the `delay` range the item waits as long as reading the message +would take, about 250 words a minute plus a few seconds for media, capped +at a minute. `mentions: true` and `reactions: true` also clear those badges +for the chat. +""" + +from __future__ import annotations + +import random +from typing import Any + +from tlgr.actions import register_action +from tlgr.actions.base import ( + Action, + ActionError, + MessageFacts, + Outcome, + Runtime, + check_keys, + require_bool, +) +from tlgr.processors import ProcessorChain + +WORDS_PER_MINUTE = 250.0 +MEDIA_READ_S = 3.0 +READING_CAP_S = 60.0 + + +def reading_seconds(facts: MessageFacts) -> float: + seconds = len(facts.text.split()) / WORDS_PER_MINUTE * 60.0 + if facts.media is not None: + seconds += MEDIA_READ_S + return min(seconds, READING_CAP_S) + + +@register_action("read") +class Read(Action): + name = "read" + cancelled_by = frozenset({"cancel", "cancel_read"}) + + def parse(self, config: Any) -> dict[str, Any]: + if config is None or config is True: + return {"mentions": False, "reactions": False} + if not isinstance(config, dict): + raise ActionError(f"{self.name}: expected {{}} or a mapping, got {config!r}") + check_keys(self.name, config, {"mentions", "reactions"}) + return { + "mentions": require_bool(self.name, "mentions", config.get("mentions", False)), + "reactions": require_bool(self.name, "reactions", config.get("reactions", False)), + } + + def plan( + self, + facts: MessageFacts, + params: dict[str, Any], + chain: ProcessorChain | None, + rng: random.Random, + ) -> list[dict[str, Any]]: + return [{"mentions": params["mentions"], "reactions": params["reactions"]}] + + def extra_delay(self, facts: MessageFacts, params: dict[str, Any]) -> float: + return reading_seconds(facts) + + def batch_key(self, item: Any) -> tuple[Any, ...] | None: + return ("read", item.chat_id, item.topic_id, item.dry_run) + + async def execute(self, items: list[Any], rt: Runtime) -> Outcome: + first = items[0] + target = max(item.msg_id for item in items) + mentions = any(item.payload.get("mentions") for item in items) + reactions = any(item.payload.get("reactions") for item in items) + if rt.read_mark(first.chat_id, first.topic_id) >= target and not (mentions or reactions): + return Outcome() + request: dict[str, Any] = {"chat": str(first.chat_id), "up_to": target} + if first.topic_id: + request["topic"] = first.topic_id + if mentions: + request["mentions"] = True + if reactions: + request["reactions"] = True + rt.begin_read(first.chat_id, first.topic_id, target) + await rt.op("message.read", request) + rt.note_read(first.chat_id, first.topic_id, target) + return Outcome() + + def describe(self, items: list[Any]) -> str: + first = items[0] + target = max(item.msg_id for item in items) + return f"read {first.chat_id} up to {target} ({len(items)} message(s))" diff --git a/tlgr/actions/reply.py b/tlgr/actions/reply.py index af903b7..6ee0af5 100644 --- a/tlgr/actions/reply.py +++ b/tlgr/actions/reply.py @@ -1,32 +1,103 @@ -"""Reply action — send a static text reply to the triggering message.""" +"""Reply action: answer the triggering message, after showing "typing...". + +The typing indicator is on by default (`typing: false` turns it off) and +lasts about as long as typing the text would take at 40 characters a second, +between 2 and 15 seconds. It runs as two steps of one pending item: the +`chat.typing` operation is started, the item is re-queued for the moment +typing ends, and then `message.send` goes out as a reply. The reply queue is +therefore never blocked for the length of somebody's typing. A broadcast +channel gets no indicator: nobody can see one there. + +Processors apply to the reply text. The text is sent as markdown, which is +what `event.reply()` did with the client's default parse mode. +""" from __future__ import annotations -import logging +import random from typing import Any from tlgr.actions import register_action -from tlgr.gateway.event import Event -from tlgr.jobs.client import JobClient +from tlgr.actions.base import ( + HOUR, + Action, + ActionError, + MessageFacts, + Outcome, + Runtime, + check_keys, + require_bool, +) from tlgr.processors import ProcessorChain -log = logging.getLogger("tlgr.actions.reply") +CHARS_PER_SECOND = 40.0 +TYPING_MIN_S = 2.0 +TYPING_MAX_S = 15.0 + + +def typing_seconds(text: str) -> float: + return max(TYPING_MIN_S, min(TYPING_MAX_S, len(text) / CHARS_PER_SECOND)) @register_action("reply") -async def action_reply( - event: Event, - config: Any, - client: JobClient, - chain: ProcessorChain | None = None, -) -> None: - if event.source != "telegram": - log.warning("reply action only supports telegram events") - return - - reply_text = str(config) if isinstance(config, str) else config.get("text", str(config)) - - if chain: - reply_text = chain.apply(reply_text) - - await event.raw.reply(reply_text) +class Reply(Action): + name = "reply" + default_expiry = 24 * HOUR + cancelled_by = frozenset({"cancel"}) + album_once = True + accepts_processors = True + + def parse(self, config: Any) -> dict[str, Any]: + typing = True + if isinstance(config, dict): + check_keys(self.name, config, {"text", "typing"}) + text = config.get("text") + if "typing" in config: + typing = require_bool(self.name, "typing", config["typing"]) + else: + text = config + if not isinstance(text, str) or not text.strip(): + raise ActionError(f"{self.name}: needs a non-empty `text`, got {text!r}") + return {"text": text, "typing": typing} + + def plan( + self, + facts: MessageFacts, + params: dict[str, Any], + chain: ProcessorChain | None, + rng: random.Random, + ) -> list[dict[str, Any]]: + text = chain.apply(params["text"]) if chain else params["text"] + if not text.strip(): + return [] + return [{"text": text, "typing": params["typing"], "typing_s": typing_seconds(text)}] + + async def execute(self, items: list[Any], rt: Runtime) -> Outcome: + item = items[0] + payload = item.payload + if item.phase == "" and payload.get("typing") and item.peer != "channel": + seconds = float(payload.get("typing_s") or TYPING_MIN_S) + request: dict[str, Any] = {"chat": str(item.chat_id), "duration": seconds} + if item.topic_id: + request["topic"] = item.topic_id + rt.spawn(rt.op("chat.typing", request), what=f"typing in {item.chat_id}") + item.phase = "send" + return Outcome(status="later", due_at=rt.now() + seconds) + rt.begin_send(item.chat_id) + result = await rt.op( + "message.send", + { + "chat": str(item.chat_id), + "text": payload["text"], + "reply_to": item.msg_id, + "parse": "md", + }, + ) + sent = (result or {}).get("id") + if sent: + rt.note_sent(item.chat_id, [int(sent)]) + return Outcome() + + def describe(self, items: list[Any]) -> str: + item = items[0] + return f"reply to msg {item.msg_id} in {item.chat_id}: {item.payload.get('text')!r}" diff --git a/tlgr/actions/view.py b/tlgr/actions/view.py new file mode 100644 index 0000000..5db7ff5 --- /dev/null +++ b/tlgr/actions/view.py @@ -0,0 +1,94 @@ +"""View action: count a channel view, or listen to a voice or round note. + +In a broadcast channel a view is `messages.getMessagesViews(increment=true)` +for the post, which is what the official apps send as a post scrolls into +view (`message.view.get --increment`). Views coalesce per chat into one call +with every id that has come due. + +In a private chat or a group the only thing to "view" is media with an +unread mark: a voice note or a round video note is marked listened with +`readMessageContents` (the channels variant in a supergroup; +`message.read --contents` picks). Self-destructing and view-once media is +never consumed unless `include_view_once: true`, because listening to it +destroys it. A message with nothing to view counts as skipped. + +`view` and `read` are independent; neither implies the other. +""" + +from __future__ import annotations + +import random +from typing import Any + +from tlgr.actions import register_action +from tlgr.actions.base import ( + HOUR, + Action, + ActionError, + MessageFacts, + Outcome, + Runtime, + check_keys, + require_bool, +) +from tlgr.processors import ProcessorChain + +#: `getMessagesViews` and `readMessageContents` take at most this many ids. +MAX_IDS = 100 + + +@register_action("view") +class View(Action): + name = "view" + default_expiry = 24 * HOUR + cancelled_by = frozenset({"cancel", "cancel_read"}) + + def parse(self, config: Any) -> dict[str, Any]: + if config is None or config is True: + return {"include_view_once": False} + if not isinstance(config, dict): + raise ActionError(f"{self.name}: expected {{}} or a mapping, got {config!r}") + check_keys(self.name, config, {"include_view_once"}) + return { + "include_view_once": require_bool( + self.name, "include_view_once", config.get("include_view_once", False) + ) + } + + def plan( + self, + facts: MessageFacts, + params: dict[str, Any], + chain: ProcessorChain | None, + rng: random.Random, + ) -> list[dict[str, Any]]: + if facts.service: + return [] + if facts.peer == "channel": + return [{"mode": "views"}] + if facts.voice_or_round and (not facts.view_once or params["include_view_once"]): + return [{"mode": "contents"}] + return [] + + def batch_key(self, item: Any) -> tuple[Any, ...] | None: + return ("view", item.chat_id, item.payload.get("mode"), item.dry_run) + + async def execute(self, items: list[Any], rt: Runtime) -> Outcome: + first = items[0] + ids = sorted({item.msg_id for item in items})[:MAX_IDS] + if first.payload.get("mode") == "views": + await rt.op( + "message.view.get", + {"chat": str(first.chat_id), "msg_id": [str(i) for i in ids], "increment": True}, + ) + else: + await rt.op( + "message.read", {"chat": str(first.chat_id), "contents": [str(i) for i in ids]} + ) + return Outcome() + + def describe(self, items: list[Any]) -> str: + first = items[0] + ids = sorted({item.msg_id for item in items}) + what = "count views of" if first.payload.get("mode") == "views" else "listen to" + return f"{what} {ids} in {first.chat_id}" diff --git a/tlgr/cli/gen.py b/tlgr/cli/gen.py index f52a6aa..f6c36ec 100644 --- a/tlgr/cli/gen.py +++ b/tlgr/cli/gen.py @@ -687,4 +687,40 @@ def build_click_tree( if path == spec.path: continue _place(root, path, build_command(spec)) + for spec in specs: + if "group-default" in spec.tags: + _default_leaf(root, spec.path) return root + + +def _default_leaf(root: dict[str, Any], path: Sequence[str]) -> None: + """Let the group above *path* run it when no subcommand is given. + + `tlgr job queue` lists the queue and `tlgr job queue cancel` cancels from + it. The registry cannot spell that with ids (an id may not also be a + group), so the list op is `job.queue.list`, tagged `group-default`, and + its group invokes it when called bare. Options still go on the leaf: + `tlgr job queue list --job dm-ack`. + """ + *groups, leaf = path + container: Any = root + for name in groups: + container = ( + container.get(name) if isinstance(container, dict) else container.commands.get(name) + ) + if not isinstance(container, click.Group): + return + command = container.commands.get(leaf) + if command is None: + return + group = container + + @click.pass_context + def run_default(ctx: click.Context) -> None: + if ctx.invoked_subcommand is None: + ctx.invoke(command) + + group.invoke_without_command = True + group.no_args_is_help = False + group.callback = run_default + group.help = f"{group.help or ''} Without a subcommand: {leaf}.".strip() diff --git a/tlgr/daemon/app.py b/tlgr/daemon/app.py index ffeb0eb..7a9bd76 100644 --- a/tlgr/daemon/app.py +++ b/tlgr/daemon/app.py @@ -102,6 +102,10 @@ def __init__( self.webhook_config = load_webhook_config(self.paths.base) self.webhook = WebhookPusher(self.webhook_config, self.paths.base, accounts=self.get_client) self._job_runner = JobRunner() + # One action scheduler per account, shared by every job on it: the + # pacer queues are per (account, action kind), not per job. + self._schedulers: dict[str, Any] = {} + self._pacing: dict[str, Any] = {} # Long file transfers: the ones `--background` hands over, and the # per-DC budgets that stop one 2 GB download from starving five # thumbnail fetches (§6.7). @@ -221,28 +225,84 @@ def touch_ipc(self) -> None: def list_jobs(self) -> list[dict[str, Any]]: return self._job_runner.list_jobs() + @property + def schedulers(self) -> dict[str, Any]: + return dict(self._schedulers) + + def action_scheduler(self, alias: str) -> Any: + """The account's job-action scheduler, created and started on first use.""" + found = self._schedulers.get(alias) + if found is not None: + return found + from tlgr.gateway.executor import DaemonOpRunner + from tlgr.gateway.pending import PendingStore + from tlgr.gateway.scheduler import ActionScheduler + + found = ActionScheduler( + alias, + DaemonOpRunner(self, alias), + store=PendingStore(self.paths.account_dir(alias) / "pending.json"), + tz=self._jobs_timezone(), + # The account-level `[presence]` setting already owns the status. + presence_enabled=self.config.presence.mode == "off", + ) + found.configure(self._pacing.get(alias)) + found.preload() + self.bus.add_handler(found.on_bus) + found.start() + self._schedulers[alias] = found + return found + + def _jobs_timezone(self) -> Any: + """`[defaults] timezone` for quiet hours; local time when it is unset.""" + name = (self.config.defaults.timezone or "").strip() + if not name: + return None + try: + from zoneinfo import ZoneInfo + + return ZoneInfo(name) + except Exception: + log.warning("unknown timezone %r in [defaults]; quiet hours use local time", name) + return None + async def remove_job(self, name: str) -> bool: - return await self._job_runner.remove_job(name) + removed = await self._job_runner.remove_job(name) + for scheduler in self._schedulers.values(): + scheduler.forget_job(name) + return removed async def enable_job(self, name: str) -> bool: return await self._job_runner.enable_job(name) async def disable_job(self, name: str) -> bool: + # Disabling says "stop": what the job had queued is dropped as well. + for scheduler in self._schedulers.values(): + scheduler.cancel(job=name) return await self._job_runner.disable_job(name) async def reload_jobs(self) -> dict[str, Any]: - from tlgr.gateway.config import load_gateway_configs - - new_configs = await asyncio.to_thread(load_gateway_configs, self.base) + from tlgr.gateway.config import load_jobs_file + + jobs_file = await asyncio.to_thread(load_jobs_file, self.base) + for problem in jobs_file.problems: + log.error("jobs.yaml: %s", problem) + new_configs = jobs_file.jobs + self._pacing = jobs_file.pacing + for alias, scheduler in self._schedulers.items(): + scheduler.configure(self._pacing.get(alias)) default_account = self.config.default_account or self.accounts.get_active() or "" old_names = set(self._job_runner._jobs) new_names = {jc.name for jc in new_configs} - removed = old_names - new_names + # A job that failed validation keeps running as it was: a typo in an + # edit must not take a working job down with it. + removed = old_names - new_names - set(jobs_file.rejected) added = new_names - old_names updated = old_names & new_names for name in removed: - await self._job_runner.remove_job(name) + await self.remove_job(name) + active: dict[str, set[str]] = {} for job_config in new_configs: if job_config.name not in added and job_config.name not in updated: continue @@ -256,11 +316,19 @@ async def reload_jobs(self) -> dict[str, Any]: log.warning("job %r references unusable account %r", job_config.name, alias) continue try: - job = self._job_runner.create_job(job_config, client, self.webhook, self.bus) + scheduler = self.action_scheduler(alias) + job = self._job_runner.create_job( + job_config, client, self.webhook, self.bus, scheduler + ) if job.enabled: job.start() + active.setdefault(alias, set()).add(job_config.name) except Exception: log.exception("could not create job %r", job_config.name) + # Pending actions saved by the last run come back once their jobs + # exist again, with the due times they had. + for alias, names in active.items(): + self._schedulers[alias].resume(names) self.activity.jobs_running = len( [j for j in self._job_runner.list_jobs() if j.get("running")] ) @@ -269,6 +337,7 @@ async def reload_jobs(self) -> dict[str, Any]: "added": sorted(added), "removed": sorted(removed), "updated": sorted(updated), + "problems": list(jobs_file.problems), } def request_shutdown(self) -> None: @@ -293,6 +362,7 @@ def v1_status(self) -> dict[str, Any]: }, "accounts": self.sessions.snapshot(), "jobs": self._job_runner.list_jobs(), + "actions": {alias: s.snapshot() for alias, s in self._schedulers.items()}, "webhook": self.webhook.snapshot(), "activity": { **{**self.activity.snapshot(), "pending_logins": self.preauth.pending_count}, @@ -412,6 +482,11 @@ async def shutdown(self, *, drain: float | None = None) -> None: with contextlib.suppress(asyncio.CancelledError, Exception): await task + # Job actions start nothing new from here; one already talking to + # Telegram is an in-flight request and the drain below waits for it. + for scheduler in list(self._schedulers.values()): + scheduler.close() + # Wait for in-flight requests rather than cancelling them: a ten # minute scan that is killed at second 599 has cost the account the # requests it already made and produced nothing (COR-11). @@ -420,6 +495,13 @@ async def shutdown(self, *, drain: float | None = None) -> None: with contextlib.suppress(Exception): await self._job_runner.stop_all() + # After the jobs (nothing new is queued) and before the sessions (a + # running action may finish): what is still pending is written to + # `accounts//pending.json` and resumes at the next start. + for scheduler in list(self._schedulers.values()): + self.bus.remove_handler(scheduler.on_bus) + with contextlib.suppress(Exception): + await scheduler.stop(timeout=min(5.0, max(0.5, deadline - time.monotonic()))) # A half-written download keeps its `.part` file, so the next # `media download --resume` continues where the shutdown stopped it. with contextlib.suppress(Exception): diff --git a/tlgr/daemon/jobs.py b/tlgr/daemon/jobs.py index c493e05..b1eb96e 100644 --- a/tlgr/daemon/jobs.py +++ b/tlgr/daemon/jobs.py @@ -24,8 +24,9 @@ def create_job( client: JobClient, webhook: WebhookPusher | None = None, bus: Any = None, + scheduler: Any = None, ) -> BaseJob: - job = Gateway(config, client, webhook, bus) + job = Gateway(config, client, webhook, bus, scheduler) self._jobs[config.name] = job return job diff --git a/tlgr/filters/README.md b/tlgr/filters/README.md index 3658815..8af415a 100644 --- a/tlgr/filters/README.md +++ b/tlgr/filters/README.md @@ -68,6 +68,17 @@ Valid message types: `text`, `photo`, `video`, `document`, `sticker`, `voice`, | `from_users` | Sender must be in list | `list[int]` | | `exclude_users` | Sender must NOT be in list | `list[int]` | +### Dialog filters (`dialog.py`) + +These may ask Telegram, so they are coroutines and only run in the job +engine (`evaluate_async`); the synchronous `evaluate` rejects them with a +reason. Both answers are cached per account. + +| Filter | Description | Value type | +|--------|-------------|------------| +| `sender_is_contact` | Sender is (or with `false`, is not) in the account's contacts (`User.contact`; refetched after 10 minutes when the update carried no sender) | `bool` | +| `chat_is_new` | In a private chat, this is the first message the dialog has ever had (one history probe per chat, then cached) | `bool` | + ## Composition Top-level filter keys are AND'd together. Use `any_of` for OR and `none_of` diff --git a/tlgr/filters/__init__.py b/tlgr/filters/__init__.py index 62ac1d0..8b08f5c 100644 --- a/tlgr/filters/__init__.py +++ b/tlgr/filters/__init__.py @@ -9,12 +9,14 @@ from __future__ import annotations -from collections.abc import Callable +from collections.abc import Awaitable, Callable from typing import Any from tlgr.gateway.event import Event -FilterFunc = Callable[[Event, Any], tuple[bool, str]] +#: A filter returns its verdict, or a coroutine of it when it has to ask +#: Telegram (`chat_is_new`); `compose.evaluate_async` awaits those. +FilterFunc = Callable[[Event, Any], tuple[bool, str] | Awaitable[tuple[bool, str]]] _REGISTRY: dict[str, FilterFunc] = {} @@ -38,5 +40,5 @@ def list_filters() -> list[str]: # Import built-in filter modules so they self-register. -from tlgr.filters import content, context, message, temporal, user # noqa: E402, F401 -from tlgr.filters.compose import evaluate, parse_filter_config # noqa: E402, F401 +from tlgr.filters import content, context, dialog, message, temporal, user # noqa: E402, F401 +from tlgr.filters.compose import evaluate, evaluate_async, parse_filter_config # noqa: E402, F401 diff --git a/tlgr/filters/compose.py b/tlgr/filters/compose.py index bf9be1a..9684149 100644 --- a/tlgr/filters/compose.py +++ b/tlgr/filters/compose.py @@ -10,6 +10,7 @@ from __future__ import annotations +import inspect from dataclasses import dataclass, field from enum import Enum from typing import Any @@ -83,7 +84,10 @@ def _parse_child_list(items: list[dict[str, Any]] | Any) -> list[FilterNode]: def evaluate(node: FilterNode | None, event: Event) -> tuple[bool, str]: """Recursively evaluate *node* against *event*. - Returns ``(passed, reason)`` just like individual filters. + Returns ``(passed, reason)`` just like individual filters. A filter that + has to ask Telegram (`chat_is_new` probes the history) is a coroutine and + can only run under `evaluate_async`; here it rejects with a reason rather + than leaking an unawaited coroutine. """ if node is None: return True, "no filters" @@ -94,7 +98,13 @@ def evaluate(node: FilterNode | None, event: Event) -> tuple[bool, str]: func = get_filter(node.filter_name) if func is None: return False, f"unknown filter: {node.filter_name}" - return func(event, node.filter_value) + result = func(event, node.filter_value) + if inspect.isawaitable(result): + close = getattr(result, "close", None) + if close is not None: + close() + return False, f"{node.filter_name} needs the job engine (it asks Telegram)" + return result if node.op is Op.AND: for child in node.children: @@ -119,3 +129,48 @@ def evaluate(node: FilterNode | None, event: Event) -> tuple[bool, str]: return True, "not-match passed" return False, "invalid node" + + +async def evaluate_async(node: FilterNode | None, event: Event) -> tuple[bool, str]: + """`evaluate`, awaiting the filters that are coroutines. + + Same short-circuit order as the sync walk, so an expensive filter placed + after a cheap one only runs for the events the cheap one let through. + """ + if node is None: + return True, "no filters" + + if node.op is Op.LEAF: + from tlgr.filters import get_filter + + func = get_filter(node.filter_name) + if func is None: + return False, f"unknown filter: {node.filter_name}" + result = func(event, node.filter_value) + if inspect.isawaitable(result): + return await result + return result + + if node.op is Op.AND: + for child in node.children: + ok, reason = await evaluate_async(child, event) + if not ok: + return False, reason + return True, "all passed" + + if node.op is Op.OR: + reasons: list[str] = [] + for child in node.children: + ok, reason = await evaluate_async(child, event) + if ok: + return True, reason + reasons.append(reason) + return False, f"none matched: {'; '.join(reasons)}" + + if node.op is Op.NOT: + ok, reason = await evaluate_async(node.children[0], event) + if ok: + return False, f"excluded: {reason}" + return True, "not-match passed" + + return False, "invalid node" diff --git a/tlgr/filters/dialog.py b/tlgr/filters/dialog.py new file mode 100644 index 0000000..1478ba3 --- /dev/null +++ b/tlgr/filters/dialog.py @@ -0,0 +1,104 @@ +"""Filters about the relationship with the other side: contact, first contact. + +Both may need Telegram. `sender_is_contact` usually reads the `contact` flag +off the sender entity the update already carried, and only fetches the user +when the update came without one. `chat_is_new` cannot be answered from the +message at all: message ids are numbered per account, not per chat, so id 1 +in a dialog does not exist and "no earlier message" takes a history probe. +Both answers are cached per account, so a busy chat costs one probe, not one +per message. +""" + +from __future__ import annotations + +import time +from collections import OrderedDict +from typing import Any + +from tlgr.filters import register_filter +from tlgr.gateway.event import Event + +#: A contact flag is trusted this long before the sender is fetched again. +CONTACT_TTL_S = 600.0 +_CACHE_SIZE = 5000 + +#: `(account, user id) -> (is a contact, when we learned it)`. +_contacts: OrderedDict[tuple[str, int], tuple[bool, float]] = OrderedDict() +#: `(account, chat id) -> the id of the first message, or 0 for "not new"`. +_first_message: OrderedDict[tuple[str, int], int] = OrderedDict() + + +def _remember(cache: OrderedDict[Any, Any], key: Any, value: Any) -> None: + cache[key] = value + cache.move_to_end(key) + while len(cache) > _CACHE_SIZE: + cache.popitem(last=False) + + +def clear_caches() -> None: + _contacts.clear() + _first_message.clear() + + +def _is_user(entity: Any) -> bool: + return type(entity).__name__ == "User" + + +@register_filter("sender_is_contact") +async def filter_sender_is_contact(event: Event, value: Any) -> tuple[bool, str]: + """The sender is (or, with `false`, is not) in the account's contacts.""" + if event.source != "telegram": + return False, "sender_is_contact requires telegram source" + message = event.raw.message + sender_id = getattr(message, "sender_id", None) + if sender_id is None or sender_id < 0: + # A channel post or an anonymous admin: no user, so no contact. + contact = False + else: + key = (event.account, int(sender_id)) + sender = getattr(message, "sender", None) + cached = _contacts.get(key) + if _is_user(sender) and not getattr(sender, "min", False): + contact = bool(getattr(sender, "contact", False)) + _remember(_contacts, key, (contact, time.monotonic())) + elif cached is not None and time.monotonic() - cached[1] < CONTACT_TTL_S: + contact = cached[0] + else: + try: + sender = await message.get_sender() + except Exception as exc: + return False, f"sender_is_contact: could not fetch the sender ({exc})" + contact = bool(getattr(sender, "contact", False)) if _is_user(sender) else False + _remember(_contacts, key, (contact, time.monotonic())) + if contact == bool(value): + return True, f"sender_is_contact={contact}" + return False, f"sender_is_contact={contact}, expected {value}" + + +@register_filter("chat_is_new") +async def filter_chat_is_new(event: Event, value: Any) -> tuple[bool, str]: + """In a private chat, this message is the first one the dialog has ever had.""" + if event.source != "telegram": + return False, "chat_is_new requires telegram source" + tg = event.raw + message = tg.message + if not getattr(tg, "is_private", False): + is_new = False + else: + key = (event.account, int(tg.chat_id)) + first = _first_message.get(key) + if first is None: + client = getattr(tg, "client", None) + if client is None: + return False, "chat_is_new: no client to probe the history with" + try: + peer = getattr(tg, "input_chat", None) or tg.chat_id + older = await client.get_messages(peer, limit=1, offset_id=int(message.id)) + except Exception as exc: + return False, f"chat_is_new: the history probe failed ({exc})" + first = 0 if older else int(message.id) + _remember(_first_message, key, first) + is_new = first != 0 and int(message.id) == first + if is_new == bool(value): + return True, f"chat_is_new={is_new}" + return False, f"chat_is_new={is_new}, expected {value}" diff --git a/tlgr/gateway/README.md b/tlgr/gateway/README.md index 30fe8f4..97bd156 100644 --- a/tlgr/gateway/README.md +++ b/tlgr/gateway/README.md @@ -10,9 +10,8 @@ flowchart TB ENV --> FILT["Filter Tree\nAND / OR / NOT composition"] FILT -->|"passed"| PROC["Processors\ntext modification chain (optional)"] PROC --> ACT["Action List\n1..N actions"] - ACT --> A1["reply"] - ACT --> A2["forward"] - ACT --> A3["future: react, pin, delete, ..."] + ACT --> SCH["Action scheduler\n(per account: delay, pacer, persistence)"] + SCH --> OPS["op layer\nforward, reply, react, read, view"] FILT -->|"rejected"| DROP["skip"] ``` @@ -62,6 +61,36 @@ jobs: processors: [add_prefix:prefix=[FWD]] # overrides job-level ``` +A DM job using the newer actions and knobs: + +```yaml +jobs: + - name: dm-ack + account: Neo + filters: {chat_type: private, sender_is_contact: true} + presence: {mode: session, quiet_hours: "01:00-08:00"} + on_takeover: cancel # cancel | cancel_read | ignore + actions: + - read: {delay: 10-90s} # + reading time + - view: {delay: 15-120s} # voice/round notes + - react: + emoji: ["👍", "❤", "🔥"] # random; or {"👍": 3, "🔥": 1} + percent: 60 + delay: 30-300s + - reply: + text: "Got it, will answer soon" + filters: {chat_is_new: true} + typing: true # default for reply + delay: 1-3m +``` + +The file is validated strictly: an unknown key on a job or an action, a bad +duration, a percent outside 0-100 or an unknown presence mode is reported +with the job's name and the action's position (`tlgr job reload +--validate-only`, `tlgr config validate`, and `tlgr job add` refuses it). At +load, a broken job is skipped and logged while the other jobs still run; on +`job reload`, a job whose edit broke it keeps running in its last good form. + ### Action syntax Actions use concise syntax -- the action name is the key: @@ -159,6 +188,13 @@ Valid types: `text`, `photo`, `video`, `document`, `sticker`, `voice`, `video_no | `from_users` | Sender must be in list | `list[int]` | | `exclude_users` | Sender must NOT be in list | `list[int]` | +#### Dialog (`dialog.py`) + +| Filter | Description | Value | +|--------|-------------|-------| +| `sender_is_contact` | Sender is in the account's contacts | `bool` | +| `chat_is_new` | First message ever in a private chat (one history probe per chat, cached) | `bool` | + ### Adding a custom filter ```python @@ -232,52 +268,142 @@ def uppercase(text, config=None): ## Actions -### Built-in actions - -#### reply - -Sends a static text reply to the triggering message. +Five built-in actions: `forward`, `reply`, `react`, `read` and `view`. Every +one runs through the op layer (the same code paths the CLI uses), in process, +for the job's account, so a job inherits the policy allow/deny list, the rate +limiter, the flood-wait budget and the self-origin events on the bus. The +reference for each action and its keys is `tlgr/actions/README.md`. ```yaml -- reply: "hello!" +actions: + - forward: {to: ["@archive"], drop_author: true} + - reply: {text: "Got it", typing: true, delay: 1-3m} + - react: {emoji: ["👍", "❤", "🔥"], percent: 60, delay: 30-300s} + - read: {delay: 10-90s, mentions: true} + - view: {delay: 15-120s} ``` -#### forward - -Forwards the message to one or more destinations. +### Knobs every action takes -```yaml -- forward: - to: ["@clean_feed", "@archive"] - drop_author: true - processors: [strip_formatting] -``` +Set on an action, or on the job as a default for all its actions (the +action's value wins). The defaults keep a job doing exactly what it did +before the knobs existed, apart from pacing. -| Key | Type | Description | -|-----|------|-------------| -| `to` | `str` or `list[str]` | Destination chat(s) | -| `drop_author` | `bool` | Remove original author | -| `processors` | `list` | Override job-level processors | -| `filters` | `dict` | AND'd with job-level filters | +| Knob | Default | Meaning | +|------|---------|---------| +| `delay` | none | Uniform random delay, measured from when the daemon received the event (not the message date). `10-90s`, `1-3m`, `5s`, `500ms`. | +| `percent` | `100` | Chance (0-100) of acting on a given message, rolled per message per action. An album is one roll. A roll-out counts as `skipped`. | +| `presence` | `leave` | `leave`, `blip`, `session`, or `{mode: ..., quiet_hours: "01:00-08:00"}`. See below. | +| `on_takeover` | `cancel` | `cancel`, `cancel_read` or `ignore`. See below. | +| `dry_run` | `false` | Run filters, rolls and scheduling, log what would be done and count it, never call Telegram. | +| `filters` | none | Per-action filters, AND'd with the job's. | -### Adding a custom action +A delay never blocks the bus: the job's handler queues the action and +returns, and the account's scheduler runs it when it is due. Actions of one +job are scheduled independently, so a forward can go at once while a +reaction on the same message waits two minutes. -```python -from tlgr.actions import register_action +### Pacing +One pacer queue per (account, action kind), shared by every job on the +account, so a backlog of reactions never delays a forward. `every` is a +floor: jitter only lengthens the gap, drawing it from `[every, 1.5 x every]`. +A backlog (hundreds of messages replayed by catch-up) is acted on, never +skipped, at the configured rate. -@register_action("react") -async def action_react(event, config, client, chain=None): - if event.source != "telegram": - return - emoji = str(config) - await client.react_to_message(event.raw.chat_id, event.raw.message.id, emoji) +```yaml +pacing: # optional; these are the defaults + Neo: # account alias + react: {every: 4s, per_hour: 300} + read: {every: 2s} + view: {every: 2s} + forward: {every: 1.5s} + reply: {every: 1.5s} + expire: {react: 24h, view: 24h, reply: 24h} # read and forward: never ``` -```yaml -- react: "thumbs_up" +An absent `per_hour` keeps the default cap; `per_hour: null` lifts it. +`expire` takes a duration or `never`; an action still pending that long +after its event arrived is dropped and counted as `expired`. + +Failures: a FLOOD_WAIT reschedules the item after the wait and doubles that +queue's spacing (up to 8x, recovering after ten quiet minutes); a transient +failure (network, server) is retried after about 5 s, 30 s and 2 min, and a +forward or read (which never expire) then every ten minutes for about an +hour and a half; any +other error (REACTION_INVALID, MESSAGE_ID_INVALID, CHAT_WRITE_FORBIDDEN, a +policy refusal) is counted under the action's `errors` and not retried. + +### Persistence + +Pending actions (delayed, or waiting for their pacer slot) are kept in +`~/.tlgr/accounts//pending.json` (mode 0600, written atomically, at +most 10 000 items) and survive a restart or a crash. At boot, once the jobs +are running, each resumes at its original due time; an overdue one goes +through the pacer. Items of a job that no longer exists are dropped. The +file also remembers recently finished items, so an update replayed after a +crash is not acted on twice. + +### Presence + +Telethon never sends `account.updateStatus`, so tlgr reads as offline while +it reacts and replies. `presence` changes that, per account: + +* `leave` (default): never touch presence. +* `blip`: online just before the action, offline about 5 s after. + Overlapping blips merge. +* `session`: online while any action of the account is running or due within + a few seconds, offline about 5 s after the last. +* `quiet_hours: "01:00-08:00"`: read, view, react and reply (not forward) are + held until the window ends, then released through the pacer, spread over + five minutes. The window is in `[defaults] timezone`, or local time. + +Two jobs asking for different modes on one account: the most online request +wins while its actions run. A `leave` action never turns presence on, and +never ends an online stretch a `blip` or `session` action started. If the +account-level `[presence] mode` is not `off`, the daemon already owns the +account's status and job presence does nothing. + +### Manual takeover + +When you act in a chat yourself from another device (read it elsewhere, or +send a message there), pending actions for messages up to that point are +dropped and counted as `superseded`, according to `on_takeover`: + +| `on_takeover` | Drops | +|---------------|-------| +| `cancel` (default) | read, view, react and reply | +| `cancel_read` | read and view only | +| `ignore` | nothing | + +Forwards are never cancelled. Reads and sends tlgr made itself are told apart +by the message ids it recorded, so its own read receipt does not count as a +takeover. + +### Inspecting and controlling + +```bash +tlgr job list # per-job and per-action counters +tlgr job get dm-ack # one job's pipeline and counters +tlgr job queue # pending actions (same as: tlgr job queue list) +tlgr job queue list --job dm-ack --chat @alice +tlgr job queue cancel 3f2a9c1b7e # by id +tlgr job queue cancel --chat @alice --yes +tlgr job queue cancel --all --yes ``` +Counters per action: `done`, `skipped` (percent roll, nothing to view, album +sibling), `superseded` (takeover or cancel), `expired`, `pending`, `errors` +(with `last_error`). `tlgr daemon status` also shows each account's queue. + +### Adding a custom action + +A built-in action is an `Action` subclass with a `plan` (runs on the bus +lane, returns payloads) and an `execute` (runs when due, calls operations +through the runtime). See `tlgr/actions/README.md`. A plain async function +registered with `@register_action` still works: it runs at once, outside the +scheduler, with `(event, config, client, chain)`. + ## Engine lifecycle ```mermaid @@ -290,19 +416,24 @@ flowchart LR The `Gateway` class extends `BaseJob`, integrating with the daemon's `JobRunner` lifecycle. On each incoming event: 1. Wrap in `Event` envelope -2. Evaluate the filter tree +2. Evaluate the filter tree (awaiting the filters that ask Telegram) 3. If passed, iterate over actions -4. For each action: check per-action filters, resolve processor chain, execute +4. For each action: check per-action filters, roll `percent`, plan the + payloads (processors applied, emoji picked) and queue them on the + account's scheduler, which runs them through the op layer when due ## Managing jobs ```bash -tlgr job add # open jobs.yaml in $EDITOR -tlgr job list # show jobs and status +tlgr job add --name dm --action 'read:delay=10-90s' --action 'react:emoji=👍' +tlgr job add --edit # open jobs.yaml in $EDITOR +tlgr job list # show jobs, status and counters +tlgr job queue # pending actions tlgr job enable -tlgr job disable +tlgr job disable # also drops what the job had queued tlgr job remove -tlgr config validate # check YAML + validate names against registries +tlgr job reload --validate-only +tlgr config validate # check YAML, knobs, pacing and names against registries ``` No code changes needed for new jobs -- the Gateway engine handles any combination of registered filters, processors, and actions. diff --git a/tlgr/gateway/config.py b/tlgr/gateway/config.py index 2dbad65..4d49aa6 100644 --- a/tlgr/gateway/config.py +++ b/tlgr/gateway/config.py @@ -1,7 +1,15 @@ """YAML-based job configuration for the Gateway pipeline. Parses ``~/.tlgr/jobs.yaml`` into :class:`GatewayConfig` objects that the -:class:`~tlgr.gateway.engine.Gateway` consumes. +:class:`~tlgr.gateway.engine.Gateway` consumes, plus the optional top-level +``pacing:`` block (per-account pacer rules and expiry). + +Validation is strict about what it understands. An unknown key on a job or +an action, a bad duration, a percent outside 0-100 or an unknown presence +mode is reported with the job's name and the action's position, rather than +being ignored into a job that silently never does what it says. A broken +job is skipped (and reported); the other jobs in the file still load, so a +typo in a new DM job cannot stop a working forward. """ from __future__ import annotations @@ -13,6 +21,8 @@ from tlgr.core.config import CONFIG_DIR from tlgr.core.errors import ConfigurationError from tlgr.filters.compose import FilterNode, parse_filter_config +from tlgr.gateway.knobs import KNOB_KEYS, KnobError, parse_duration, parse_knobs +from tlgr.gateway.pacer import DEFAULT_PACING, PacingRule from tlgr.processors import ProcessorChain, create_chain_from_list @@ -24,6 +34,10 @@ class ActionConfig: config: Any = None filters: FilterNode | None = None processors: ProcessorChain | None = None + #: The action's own config, validated by the action (None: not parsed yet). + params: dict[str, Any] | None = None + #: Knobs set on this action, parsed; job knobs fill in the rest. + knobs: dict[str, Any] = field(default_factory=dict) ALL_EVENT_TYPES = frozenset( @@ -37,6 +51,11 @@ class ActionConfig: } ) +JOB_KEYS = frozenset( + {"name", "account", "enabled", "events", "filters", "processors", "actions"} | KNOB_KEYS +) +_PIPELINE_KEYS = frozenset({"filters", "processors"}) + @dataclass class GatewayConfig: @@ -49,9 +68,32 @@ class GatewayConfig: filters: FilterNode | None = None processors: ProcessorChain | None = None actions: list[ActionConfig] = field(default_factory=list) + #: Job-level knobs: defaults for every action of the job. + knobs: dict[str, Any] = field(default_factory=dict) + + +class JobConfigError(ConfigurationError): + """One job (or the pacing block) that cannot be loaded; `problems` says why.""" + + def __init__(self, problems: list[str]) -> None: + super().__init__("; ".join(problems)) + self.problems = problems + +def _split_action(name: str, value: Any) -> tuple[Any, dict[str, Any], Any, Any]: + """`(the action's own config, knob keys, filters, processors)`. -def _parse_action(raw: dict[str, Any]) -> ActionConfig: + A react written as a weighted mapping (`{"👍": 3}`) is all config; any + other mapping carries its knobs and pipeline keys beside its own keys. + """ + if not isinstance(value, dict) or (name == "react" and "emoji" not in value): + return value, {}, None, None + own = {k: v for k, v in value.items() if k not in KNOB_KEYS and k not in _PIPELINE_KEYS} + knobs = {k: value[k] for k in KNOB_KEYS if k in value} + return own, knobs, value.get("filters"), value.get("processors") + + +def _parse_action(raw: dict[str, Any], *, where: str = "action") -> ActionConfig: """Parse a concise action entry. Concise syntax: the action name is the dict key, the value is its config. @@ -60,34 +102,70 @@ def _parse_action(raw: dict[str, Any]) -> ActionConfig: {"reply": "hello"} -> ActionConfig(name="reply", config="hello") {"forward": {"to": "@chan"}} -> ActionConfig(name="forward", config={"to": "@chan"}) + {"react": ["👍", "🔥"]} -> ActionConfig(name="react", config=["👍", "🔥"]) """ - for key, value in raw.items(): - ac = ActionConfig(name=key) - - if isinstance(value, str): - ac.config = value - elif isinstance(value, dict): - ac.config = {k: v for k, v in value.items() if k not in ("filters", "processors")} - if len(ac.config) == 1 and "text" in ac.config: - ac.config = ac.config["text"] - ac.filters = parse_filter_config(value.get("filters")) - procs = value.get("processors") - if procs: - ac.processors = create_chain_from_list(procs) if isinstance(procs, list) else None + from tlgr.actions import get_action, get_builtin + + if not isinstance(raw, dict) or len(raw) != 1: + raise JobConfigError([f"{where}: an action is a mapping with exactly one key"]) + name, value = next(iter(raw.items())) + name = str(name) + own, knob_raw, filters_raw, procs = _split_action(name, value) + problems: list[str] = [] + + ac = ActionConfig(name=name) + # The old shape, kept for actions written against the function interface: + # a mapping with only `text` collapses to the text. + ac.config = own["text"] if isinstance(own, dict) and set(own) == {"text"} else own + + found = get_action(name) + if found is None: + problems.append(f"{where}: unknown action {name!r}") + try: + ac.knobs = parse_knobs(knob_raw, where=where) + except KnobError as exc: + problems.append(str(exc)) + if filters_raw is not None and not isinstance(filters_raw, dict): + problems.append(f"{where}: filters must be a mapping") + else: + ac.filters = parse_filter_config(filters_raw) + if procs is not None: + if not isinstance(procs, list): + problems.append(f"{where}: processors must be a list") else: - ac.config = value - - return ac - - return ActionConfig() + builtin = get_builtin(name) + if builtin is not None and not builtin.accepts_processors: + problems.append(f"{where}: {name} does not take processors") + try: + ac.processors = create_chain_from_list(procs) if procs else None + except ValueError as exc: + problems.append(f"{where}: {exc}") + builtin = get_builtin(name) + if builtin is not None: + try: + ac.params = builtin.parse(own) + except KnobError as exc: + problems.append(f"{where}: {exc}") + if problems: + raise JobConfigError(problems) + return ac def _parse_job(raw: dict[str, Any]) -> GatewayConfig: - cfg = GatewayConfig( - name=raw.get("name", ""), - account=raw.get("account", ""), - enabled=raw.get("enabled", True), - ) + """One `jobs:` entry; raises `JobConfigError` listing every problem in it.""" + name = raw.get("name", "") + label = f"job {name!r}" if name else "a job without a name" + problems: list[str] = [] + if not name: + problems.append(f"{label}: `name` is required") + unknown = sorted(set(raw) - JOB_KEYS) + if unknown: + problems.append(f"{label}: unknown key(s) {unknown}; valid: {', '.join(sorted(JOB_KEYS))}") + enabled = raw.get("enabled", True) + if not isinstance(enabled, bool): + problems.append(f"{label}: enabled must be true or false") + cfg = GatewayConfig(name=str(name), account=str(raw.get("account", "") or "")) + cfg.enabled = bool(enabled) raw_events = raw.get("events") if raw_events and isinstance(raw_events, list): @@ -95,26 +173,162 @@ def _parse_job(raw: dict[str, Any]) -> GatewayConfig: elif raw_events and isinstance(raw_events, str): cfg.events = [raw_events] if raw_events in ALL_EVENT_TYPES else ["new_message"] - cfg.filters = parse_filter_config(raw.get("filters")) + filters = raw.get("filters") + if filters is not None and not isinstance(filters, dict): + problems.append(f"{label}: filters must be a mapping") + else: + cfg.filters = parse_filter_config(filters) procs = raw.get("processors") if procs and isinstance(procs, list): - cfg.processors = create_chain_from_list(procs) + try: + cfg.processors = create_chain_from_list(procs) + except ValueError as exc: + problems.append(f"{label}: {exc}") - actions_raw = raw.get("actions", []) - for action_raw in actions_raw: - if isinstance(action_raw, dict): - cfg.actions.append(_parse_action(action_raw)) + try: + cfg.knobs = parse_knobs(raw, where=label) + except KnobError as exc: + problems.append(str(exc)) + actions_raw = raw.get("actions", []) + if not isinstance(actions_raw, list): + problems.append(f"{label}: actions must be a list") + actions_raw = [] + for position, action_raw in enumerate(actions_raw, start=1): + try: + cfg.actions.append(_parse_action(action_raw, where=f"{label}, action {position}")) + except JobConfigError as exc: + problems.extend(exc.problems) + + if problems: + raise JobConfigError(problems) return cfg -def load_gateway_configs(base: Path | None = None) -> list[GatewayConfig]: - """Load all gateway jobs from ``jobs.yaml``.""" +def parse_pacing(raw: Any) -> dict[str, Any]: + """The top-level `pacing:` block → `{alias: AccountPacing}`.""" + from tlgr.gateway.scheduler import AccountPacing + + if raw is None: + return {} + if not isinstance(raw, dict): + raise JobConfigError(["pacing: must be a mapping of account alias to settings"]) + problems: list[str] = [] + out: dict[str, Any] = {} + kinds = set(DEFAULT_PACING) + for alias, block in raw.items(): + where = f"pacing.{alias}" + if not isinstance(block, dict): + problems.append(f"{where}: must be a mapping") + continue + pacing = AccountPacing() + for key, value in block.items(): + if key == "expire": + if not isinstance(value, dict): + problems.append(f"{where}.expire: must be a mapping of action to duration") + continue + for kind, duration in value.items(): + if kind not in kinds: + problems.append(f"{where}.expire: unknown action {kind!r}") + continue + if duration in (None, "never", 0): + pacing.expire[kind] = None + continue + try: + pacing.expire[kind] = parse_duration( + duration, what=f"{where}.expire.{kind}" + ) + except KnobError as exc: + problems.append(str(exc)) + continue + if key not in kinds: + problems.append( + f"{where}: unknown key {key!r}; use {', '.join(sorted(kinds))} or expire" + ) + continue + if not isinstance(value, dict) or set(value) - {"every", "per_hour"}: + problems.append(f"{where}.{key}: takes {{every: , per_hour: }}") + continue + default = DEFAULT_PACING[key] + try: + every = ( + parse_duration(value["every"], what=f"{where}.{key}.every") + if "every" in value + else default.every + ) + except KnobError as exc: + problems.append(str(exc)) + continue + # An absent `per_hour` keeps the default cap; `per_hour: null` lifts it. + per_hour = value.get("per_hour", default.per_hour) + if per_hour is not None and ( + isinstance(per_hour, bool) or not isinstance(per_hour, int) or per_hour <= 0 + ): + problems.append(f"{where}.{key}.per_hour: must be a positive whole number") + continue + pacing.rules[key] = PacingRule(every=every, per_hour=per_hour) + out[str(alias)] = pacing + if problems: + raise JobConfigError(problems) + return out + + +@dataclass +class JobsFile: + """Everything `jobs.yaml` says, and every problem found reading it.""" + + jobs: list[GatewayConfig] = field(default_factory=list) + pacing: dict[str, Any] = field(default_factory=dict) + problems: list[str] = field(default_factory=list) + #: Names of jobs that are in the file but failed validation. + rejected: list[str] = field(default_factory=list) + + +def parse_jobs_document(data: Any) -> JobsFile: + out = JobsFile() + if not data: + return out + if not isinstance(data, dict): + out.problems.append("jobs.yaml must be a mapping with a `jobs:` list") + return out + unknown = sorted(set(data) - {"jobs", "pacing"}) + if unknown: + out.problems.append(f"jobs.yaml: unknown top-level key(s) {unknown}; use jobs, pacing") + try: + out.pacing = parse_pacing(data.get("pacing")) + except JobConfigError as exc: + out.problems.extend(exc.problems) + jobs = data.get("jobs") or [] + if not isinstance(jobs, list): + out.problems.append("jobs.yaml: `jobs` must be a list") + return out + seen: set[str] = set() + for entry in jobs: + if not isinstance(entry, dict): + out.problems.append("jobs.yaml: every entry under `jobs` must be a mapping") + continue + try: + config = _parse_job(entry) + except JobConfigError as exc: + out.problems.extend(exc.problems) + if entry.get("name"): + out.rejected.append(str(entry["name"])) + continue + if config.name in seen: + out.problems.append(f"job {config.name!r}: the name is used twice") + continue + seen.add(config.name) + out.jobs.append(config) + return out + + +def load_jobs_file(base: Path | None = None) -> JobsFile: + """Read and validate ``jobs.yaml``; a missing file is an empty one.""" base = base or CONFIG_DIR jobs_path = base / "jobs.yaml" if not jobs_path.exists(): - return [] + return JobsFile() try: import yaml @@ -125,11 +339,12 @@ def load_gateway_configs(base: Path | None = None) -> list[GatewayConfig]: with open(jobs_path) as f: data = yaml.safe_load(f) + return parse_jobs_document(data) - if not data or "jobs" not in data: - return [] - return [_parse_job(j) for j in data["jobs"] if isinstance(j, dict)] +def load_gateway_configs(base: Path | None = None) -> list[GatewayConfig]: + """Load all valid gateway jobs from ``jobs.yaml``.""" + return load_jobs_file(base).jobs def save_gateway_configs(configs: list[GatewayConfig], base: Path | None = None) -> None: diff --git a/tlgr/gateway/engine.py b/tlgr/gateway/engine.py index e6f7a7a..b473138 100644 --- a/tlgr/gateway/engine.py +++ b/tlgr/gateway/engine.py @@ -20,6 +20,13 @@ Without a bus — a unit test, or a daemon that has not started one — the job falls back to registering Telethon handlers exactly as v1 did. + +Actions no longer run inside the handler. The job evaluates its filters, +rolls each action's `percent`, plans the payloads and hands them to the +account's `ActionScheduler` (`gateway/scheduler.py`), which owns delays, +pacing, persistence and execution through the op layer. The handler returns +as soon as the items are queued, so a 90 second `delay` never holds the bus +lane the event arrived on. """ from __future__ import annotations @@ -30,9 +37,12 @@ from typing import Any from tlgr.actions import get_action -from tlgr.filters.compose import FilterNode, Op, evaluate +from tlgr.actions.base import Action, MessageFacts, facts_from_event +from tlgr.filters.compose import FilterNode, Op, evaluate_async from tlgr.gateway.config import ActionConfig, GatewayConfig from tlgr.gateway.event import Event +from tlgr.gateway.knobs import merge_knobs +from tlgr.gateway.pending import PendingItem from tlgr.gateway.tlevents import build_event, builder_for_job_event from tlgr.jobs.base import BaseJob from tlgr.jobs.client import JobClient @@ -113,8 +123,10 @@ def __init__( client: JobClient, webhook=None, bus=None, + scheduler=None, ) -> None: self._gw = config + self._scheduler = scheduler shim = _GatewayJobConfig(config) super().__init__(shim, client, webhook) # type: ignore[arg-type] self._handlers: list = [] @@ -124,6 +136,14 @@ def __init__( self._refs_pending = False self._next_resolve = 0.0 self._unresolved_logged: set[str] = set() + self._no_scheduler_logged = False + + def status(self) -> dict[str, Any]: + row = super().status() + row.update(self._stats) + scheduler = self._scheduler + row["actions"] = scheduler.job_stats(self.name) if scheduler is not None else [] + return row async def setup(self) -> None: await self._resolve_chat_refs() @@ -244,19 +264,22 @@ async def _handle(self, tg_event, event_type: str = "new_message") -> None: event_type=event_type, ) - ok, reason = evaluate(self._gw.filters, envelope) + ok, reason = await evaluate_async(self._gw.filters, envelope) if not ok: self._stats["skipped"] += 1 return self._stats["matched"] += 1 - for action_cfg in self._gw.actions: - await self._run_action(action_cfg, envelope) + facts: list[MessageFacts] = [] + for index, action_cfg in enumerate(self._gw.actions): + await self._run_action(index, action_cfg, envelope, facts) - async def _run_action(self, ac: ActionConfig, envelope: Event) -> None: + async def _run_action( + self, index: int, ac: ActionConfig, envelope: Event, facts: list[MessageFacts] + ) -> None: if ac.filters: - ok, reason = evaluate(ac.filters, envelope) + ok, reason = await evaluate_async(ac.filters, envelope) if not ok: return @@ -268,8 +291,96 @@ async def _run_action(self, ac: ActionConfig, envelope: Event) -> None: chain = ac.processors or self._gw.processors + if not isinstance(func, Action): + # An action written against the old function interface runs at + # once, outside the scheduler, exactly as it always did. + try: + await func(envelope, ac.config, self.client, chain) + except Exception as e: + log.warning("[%s] action '%s' failed: %s", self.name, ac.name, e) + self._stats["errors"] += 1 + return + + scheduler = self._scheduler + if scheduler is None: + if not self._no_scheduler_logged: + log.warning("[%s] no action scheduler for this account; actions not run", self.name) + self._no_scheduler_logged = True + self._stats["errors"] += 1 + return + try: - await func(envelope, ac.config, self.client, chain) + self._schedule(scheduler, index, func, ac, chain, envelope, facts) except Exception as e: - log.warning("[%s] action '%s' failed: %s", self.name, ac.name, e) - self._stats["errors"] += 1 + log.warning("[%s] action '%s' could not be scheduled: %s", self.name, ac.name, e) + scheduler.count(self.name, index, ac.name, "errors", error=str(e)) + + def _schedule( + self, + scheduler: Any, + index: int, + action: Action, + ac: ActionConfig, + chain: Any, + envelope: Event, + facts_memo: list[MessageFacts], + ) -> None: + """Roll, plan and queue one action for one event. Never awaits.""" + if not facts_memo: + facts_memo.append(facts_from_event(envelope.raw)) + facts = facts_memo[0] + params = ac.params if ac.params is not None else action.parse(ac.config) + knobs = merge_knobs(self._gw.knobs, ac.knobs) + + album = None + if action.album_once and facts.grouped_id: + album = (self.name, index, facts.chat_id, facts.grouped_id) + rolled = scheduler.album_roll(album, knobs.percent) + else: + rolled = scheduler.roll(knobs.percent) + if not rolled: + scheduler.count(self.name, index, action.name, "skipped") + return + + payloads = action.plan(facts, params, chain, scheduler.rng) + if not payloads: + scheduler.count(self.name, index, action.name, "skipped") + return + + received = scheduler.clock.now() + # The key identifies the work: the same message replayed by catch-up + # after a crash is not acted on twice. An edit is new work, so it is + # keyed by its edit time as well. + event = envelope.event_type + if event != "new_message": + edited = getattr(envelope.raw.message, "edit_date", None) + event = f"{event}@{edited.timestamp() if edited else ''}" + expiry = scheduler.expiry_for(action) + quiet = knobs.presence.quiet_hours + for payload in payloads: + low, high = knobs.delay + delay = scheduler.rng.uniform(low, high) if high > low else low + delay += action.extra_delay(facts, params) + extra = action.key_extra(payload) + item = PendingItem( + id=scheduler.new_id(), + job=self.name, + action=action.name, + index=index, + account=self._gw.account, + chat_id=facts.chat_id, + msg_id=facts.msg_id, + received_at=received, + due_at=received + delay, + expires_at=received + expiry if expiry is not None else None, + peer=facts.peer, + topic_id=facts.topic_id, + grouped_id=facts.grouped_id, + presence=knobs.presence.mode, + quiet_hours=quiet.text if quiet is not None else None, + on_takeover=knobs.on_takeover, + dry_run=knobs.dry_run, + payload=payload, + key=f"{self.name}|{index}|{event}|{facts.chat_id}|{facts.msg_id}|{extra}", + ) + scheduler.submit(item, album=album, has_caption=bool(facts.text)) diff --git a/tlgr/gateway/executor.py b/tlgr/gateway/executor.py new file mode 100644 index 0000000..697eb3f --- /dev/null +++ b/tlgr/gateway/executor.py @@ -0,0 +1,56 @@ +"""How a job action reaches Telegram: through the op layer, in process. + +A job used to call Telethon directly (`client.forward_messages`, +`event.reply`), which meant none of what the CLI path guarantees applied to +it: no policy allow/deny, no rate limiter, no flood-wait budget, no +self-origin event on the bus, and none of the per-peer edge cases the ops +handle (`message.read` choosing between three read RPCs, `reaction.add` +reading the current set first). Running every action as an operation through +`dispatch.execute` gives a job the same guarantees as `tlgr message read` +typed by hand. +""" + +from __future__ import annotations + +import uuid +from typing import Any, Protocol + +__all__ = ["DaemonOpRunner", "OpRunner"] + +#: The longest FLOOD_WAIT a job action sleeps off inside the request. A longer +#: wait comes back as RATE_LIMITED and the scheduler reschedules the item, +#: which frees the queue instead of holding it for the whole wait. +JOB_FLOOD_WAIT_MAX = 10 + + +class OpRunner(Protocol): + async def __call__(self, op: str, request: dict[str, Any]) -> Any: ... + + +class DaemonOpRunner: + """Run an operation for one account through the daemon's dispatcher.""" + + def __init__(self, daemon: Any, account: str) -> None: + self.daemon = daemon + self.account = account + + async def __call__(self, op: str, request: dict[str, Any]) -> Any: + from tlgr.daemon import dispatch + from tlgr.models.base import to_builtins + from tlgr.models.envelope import OpRequest + from tlgr.version import PROTOCOL, VERSION + + op_request = OpRequest( + op=op, + account=self.account, + request=request, + request_id=f"job-{uuid.uuid4().hex[:12]}", + client_version=VERSION, + protocol=PROTOCOL, + flood_wait_max=JOB_FLOOD_WAIT_MAX, + ) + # Counted as a request, so the shutdown drain waits for an action + # that is half way through instead of cutting it off. + with dispatch.in_flight(self.daemon): + _, _, result = await dispatch.execute(self.daemon, op_request) + return to_builtins(result) if result is not None else None diff --git a/tlgr/gateway/knobs.py b/tlgr/gateway/knobs.py new file mode 100644 index 0000000..0a5e8f0 --- /dev/null +++ b/tlgr/gateway/knobs.py @@ -0,0 +1,223 @@ +"""The knobs every job action shares: delay, percent, presence, takeover, dry run. + +They are parsed once, when `jobs.yaml` is read, into plain values the +scheduler can use without re-reading YAML: a delay is a `(low, high)` pair of +seconds, a quiet window is two minutes-of-day. A malformed value raises +`KnobError` with a message that names the key and the value, because the only +person who will ever read it is the one who typed the YAML. + +Durations accept `500ms`, `5s`, `1.5s`, `2m`, `1h`, `24h` and a bare number of +seconds; a range is two durations with a dash, the unit written once or on +both ends (`10-90s`, `1m-3m`, `500ms-2s`). +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from datetime import datetime, timedelta +from typing import Any + +__all__ = [ + "PRESENCE_MODES", + "TAKEOVER_MODES", + "ActionKnobs", + "KnobError", + "Presence", + "QuietHours", + "merge_knobs", + "parse_delay", + "parse_duration", + "parse_knobs", + "parse_percent", + "parse_presence", +] + +PRESENCE_MODES = ("leave", "blip", "session") +TAKEOVER_MODES = ("cancel", "cancel_read", "ignore") + +#: The knob names valid on a job and on every action. +KNOB_KEYS = frozenset({"delay", "percent", "presence", "on_takeover", "dry_run"}) + +_UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0, "h": 3600.0, "d": 86400.0} +_DURATION_RE = re.compile(r"^\s*(\d+(?:\.\d+)?)\s*(ms|s|m|h|d)?\s*$", re.IGNORECASE) +_CLOCK_RE = re.compile(r"^\s*(\d{1,2}):(\d{2})\s*$") + + +class KnobError(ValueError): + """A knob value that cannot be parsed. The message is for a human.""" + + +def parse_duration(value: Any, *, what: str = "duration", default_unit: str = "s") -> float: + """`"90s"`, `"1.5m"`, `"500ms"`, `30` → seconds.""" + if isinstance(value, bool): + raise KnobError(f"{what}: {value!r} is not a duration") + if isinstance(value, (int, float)): + if value < 0: + raise KnobError(f"{what}: {value!r} is negative") + return float(value) * _UNITS[default_unit] + if not isinstance(value, str): + raise KnobError(f"{what}: {value!r} is not a duration (try 5s, 2m, 500ms)") + match = _DURATION_RE.match(value) + if match is None: + raise KnobError(f"{what}: {value!r} is not a duration (try 5s, 2m, 500ms)") + number, unit = match.groups() + return float(number) * _UNITS[(unit or default_unit).lower()] + + +def parse_delay(value: Any) -> tuple[float, float]: + """`"10-90s"` → `(10.0, 90.0)`; a single duration is a fixed delay.""" + if value is None: + return (0.0, 0.0) + if isinstance(value, str) and "-" in value.strip().lstrip("-"): + low_raw, _, high_raw = value.strip().partition("-") + high_match = _DURATION_RE.match(high_raw) + low_match = _DURATION_RE.match(low_raw) + if high_match is None or low_match is None: + raise KnobError(f"delay: {value!r} is not a range (try 10-90s or 1m-3m)") + # `10-90s`: the unit written once, on the high end, applies to both. + unit = high_match.group(2) or "s" + low = parse_duration(low_raw, what="delay", default_unit=unit.lower()) + high = parse_duration(high_raw, what="delay") + if low > high: + raise KnobError(f"delay: {value!r} has its low end above its high end") + return (low, high) + seconds = parse_duration(value, what="delay") + return (seconds, seconds) + + +def parse_percent(value: Any) -> int: + if isinstance(value, bool) or not isinstance(value, (int, float, str)): + raise KnobError(f"percent: {value!r} is not a number from 0 to 100") + try: + number = float(str(value).strip().rstrip("%")) + except ValueError as exc: + raise KnobError(f"percent: {value!r} is not a number from 0 to 100") from exc + if number != int(number) or not 0 <= number <= 100: + raise KnobError(f"percent: {value!r} must be a whole number from 0 to 100") + return int(number) + + +def _minutes(text: str, raw: Any) -> int: + match = _CLOCK_RE.match(text) + if match is None: + raise KnobError(f"quiet_hours: {raw!r} is not HH:MM-HH:MM") + hours, minutes = int(match.group(1)), int(match.group(2)) + if hours > 23 or minutes > 59: + raise KnobError(f"quiet_hours: {raw!r} has an impossible time") + return hours * 60 + minutes + + +@dataclass(frozen=True) +class QuietHours: + """A daily window, `start` inclusive and `end` exclusive, in minutes of day. + + `start > end` wraps midnight (`23:00-07:00`); `start == end` is empty. + """ + + start: int + end: int + text: str = "" + + @classmethod + def parse(cls, value: Any) -> QuietHours: + if not isinstance(value, str) or "-" not in value: + raise KnobError(f"quiet_hours: {value!r} is not HH:MM-HH:MM") + start, _, end = value.partition("-") + return cls(_minutes(start, value), _minutes(end, value), value.strip()) + + def contains(self, moment: datetime) -> bool: + minute = moment.hour * 60 + moment.minute + if self.start == self.end: + return False + if self.start < self.end: + return self.start <= minute < self.end + return minute >= self.start or minute < self.end + + def window_end(self, moment: datetime) -> datetime: + """When the window that contains *moment* closes (*moment* if it is outside).""" + if not self.contains(moment): + return moment + end = moment.replace(hour=self.end // 60, minute=self.end % 60, second=0, microsecond=0) + if end <= moment: + end += timedelta(days=1) + return end + + +@dataclass(frozen=True) +class Presence: + mode: str = "leave" + quiet_hours: QuietHours | None = None + + +def parse_presence(value: Any) -> Presence: + """`session`, or `{mode: session, quiet_hours: "01:00-08:00"}`.""" + if value is None: + return Presence() + if isinstance(value, str): + mode, quiet = value.strip().lower(), None + elif isinstance(value, dict): + unknown = set(value) - {"mode", "quiet_hours"} + if unknown: + raise KnobError(f"presence: unknown key(s) {sorted(unknown)}; use mode, quiet_hours") + mode = str(value.get("mode", "leave")).strip().lower() + quiet = value.get("quiet_hours") + else: + raise KnobError(f"presence: {value!r} is not a mode or a mapping") + if mode not in PRESENCE_MODES: + raise KnobError(f"presence: unknown mode {mode!r}; use one of {', '.join(PRESENCE_MODES)}") + return Presence(mode=mode, quiet_hours=QuietHours.parse(quiet) if quiet else None) + + +@dataclass(frozen=True) +class ActionKnobs: + """The resolved knobs for one action, job defaults already applied. + + The defaults reproduce what a job did before the knobs existed: act at + once, every time, without touching presence or the dry-run switch. + """ + + delay: tuple[float, float] = (0.0, 0.0) + percent: int = 100 + presence: Presence = field(default_factory=Presence) + on_takeover: str = "cancel" + dry_run: bool = False + + +def parse_knobs(raw: dict[str, Any], *, where: str = "") -> dict[str, Any]: + """The knob keys present in *raw*, parsed. Absent keys stay absent. + + Partial on purpose: a job's knobs are defaults for its actions, so "the + action did not say" must stay distinguishable from "the action said the + default". + """ + prefix = f"{where}: " if where else "" + out: dict[str, Any] = {} + try: + if "delay" in raw: + out["delay"] = parse_delay(raw["delay"]) + if "percent" in raw: + out["percent"] = parse_percent(raw["percent"]) + if "presence" in raw: + out["presence"] = parse_presence(raw["presence"]) + if "on_takeover" in raw: + mode = str(raw["on_takeover"]).strip().lower() + if mode not in TAKEOVER_MODES: + raise KnobError( + f"on_takeover: unknown mode {raw['on_takeover']!r}; " + f"use one of {', '.join(TAKEOVER_MODES)}" + ) + out["on_takeover"] = mode + if "dry_run" in raw: + if not isinstance(raw["dry_run"], bool): + raise KnobError(f"dry_run: {raw['dry_run']!r} is not true or false") + out["dry_run"] = raw["dry_run"] + except KnobError as exc: + raise KnobError(f"{prefix}{exc}") from None + return out + + +def merge_knobs(job: dict[str, Any], action: dict[str, Any]) -> ActionKnobs: + """Job knobs as defaults, the action's own knobs on top.""" + merged = {**job, **action} + return ActionKnobs(**merged) diff --git a/tlgr/gateway/pacer.py b/tlgr/gateway/pacer.py new file mode 100644 index 0000000..f1a33b4 --- /dev/null +++ b/tlgr/gateway/pacer.py @@ -0,0 +1,125 @@ +"""Pacing for job actions: one queue per (account, action kind). + +The daemon's token buckets (`daemon/ratelimit.py`) protect the account from a +burst of requests. They do not make a stream of automated reactions look like +a person: a bucket with a burst of twenty lets twenty reactions out in the same +second after a quiet hour. A pacer is the other half. It spaces one kind of +action on one account at a floor of `every` seconds plus jitter, and caps the +count in any rolling hour. + +Jitter only ever lengthens the gap. `every: 4s` is a promise that reactions +are never closer than four seconds, so the spacing is drawn from +`[every, every * 1.5]` (a mean of 1.25x, about +-20% around it) rather than +from a window centred on `every` that would break the floor. + +A FLOOD_WAIT on a kind pushes that kind's next slot past the wait and doubles +its spacing, up to eight times. The slow-down lifts once ten minutes pass +without another one. Other kinds are unaffected: a flood on reactions says +nothing about forwards. + +Time comes from an injected clock (`now()`, wall seconds), so tests drive the +spacing without sleeping. +""" + +from __future__ import annotations + +import random +from collections import deque +from dataclasses import dataclass +from typing import Protocol + +__all__ = ["DEFAULT_PACING", "Clock", "Pacer", "PacingRule", "SystemClock"] + + +class Clock(Protocol): + def now(self) -> float: ... + + async def sleep(self, seconds: float) -> None: ... + + +class SystemClock: + """Wall-clock time and real sleeps.""" + + def now(self) -> float: + import time + + return time.time() + + async def sleep(self, seconds: float) -> None: + import asyncio + + await asyncio.sleep(max(0.0, seconds)) + + +@dataclass(frozen=True) +class PacingRule: + every: float + per_hour: int | None = None + + +#: Cautious defaults, per account. A reaction every four seconds at most and +#: three hundred an hour is well under anything a person does by hand. +DEFAULT_PACING: dict[str, PacingRule] = { + "react": PacingRule(every=4.0, per_hour=300), + "read": PacingRule(every=2.0), + "view": PacingRule(every=2.0), + "forward": PacingRule(every=1.5), + "reply": PacingRule(every=1.5), +} + +#: The spacing multiplier is drawn from this range. +JITTER = (1.0, 1.5) +MAX_SLOWDOWN = 8.0 +SLOWDOWN_RECOVERY_S = 600.0 +HOUR = 3600.0 + + +class Pacer: + """Spacing and an hourly cap for one action kind on one account.""" + + def __init__(self, rule: PacingRule, *, rng: random.Random | None = None) -> None: + self.rule = rule + self.rng = rng or random.Random() + self.next_at = 0.0 + self.slowdown = 1.0 + self.last_flood_at = 0.0 + self._recent: deque[float] = deque() + + def configure(self, rule: PacingRule) -> None: + self.rule = rule + + def delay(self, now: float) -> float: + """Seconds until the next slot opens; 0 when one is open now.""" + if self.slowdown > 1.0 and now - self.last_flood_at >= SLOWDOWN_RECOVERY_S: + self.slowdown = 1.0 + wait = max(0.0, self.next_at - now) + cap = self.rule.per_hour + if cap: + while self._recent and now - self._recent[0] >= HOUR: + self._recent.popleft() + if len(self._recent) >= cap: + # A millisecond past the hour, so float rounding cannot land the + # slot inside the window it is waiting out. + wait = max(wait, self._recent[0] + HOUR - now + 0.001) + return wait + + def consume(self, now: float) -> None: + """Take the open slot and schedule the next one.""" + spacing = self.rule.every * self.rng.uniform(*JITTER) * self.slowdown + self.next_at = now + spacing + if self.rule.per_hour: + self._recent.append(now) + + def penalize(self, now: float, wait: float) -> None: + """A FLOOD_WAIT of *wait* seconds: hold the queue and widen its spacing.""" + self.slowdown = min(MAX_SLOWDOWN, self.slowdown * 2.0) + self.last_flood_at = now + self.next_at = max(self.next_at, now + max(0.0, wait)) + + def snapshot(self) -> dict[str, float | int | None]: + return { + "every_s": self.rule.every, + "per_hour": self.rule.per_hour, + "slowdown": self.slowdown, + "last_hour": len(self._recent), + } diff --git a/tlgr/gateway/pending.py b/tlgr/gateway/pending.py new file mode 100644 index 0000000..3be4c88 --- /dev/null +++ b/tlgr/gateway/pending.py @@ -0,0 +1,127 @@ +"""Pending job actions, and the file that keeps them across a restart. + +An action that is waiting (for its delay, for its pacer slot, for quiet hours +to end) is a `PendingItem`. It carries everything needed to run it again +without the Telethon event that produced it: the chat, the message, the album +it belongs to, the forum topic, and the action's own payload (the emoji, the +reply text after processors, the forward destination). A restart therefore +loses nothing: the daemon reloads the file and each item resumes with its +absolute due time, so an item due at 10:03 still runs at 10:03, and one that +fell due while the daemon was down runs as soon as its pacer allows. + +The file is `accounts//pending.json`, written with `write_private` +(a temp file, chmod 0600, fsync, rename), so a crash mid-write leaves the +previous version rather than half of a new one. It also keeps the keys of +recently finished items: an update replayed after a crash, before the update +state was saved, is recognised and not acted on twice. +""" + +from __future__ import annotations + +import contextlib +import json +import logging +from dataclasses import asdict, dataclass, field, fields +from pathlib import Path +from typing import Any + +log = logging.getLogger("tlgr.gateway.pending") + +__all__ = ["MAX_PENDING", "PendingItem", "PendingStore"] + +#: Per account. A backlog this deep means something is wrong upstream; the +#: next item is refused (and counted as an error) rather than growing the file +#: without bound. +MAX_PENDING = 10_000 +#: How many finished-item keys are remembered for replay de-duplication. +RECENT_KEYS = 5_000 + +_VERSION = 1 + + +@dataclass +class PendingItem: + id: str + job: str + action: str + index: int + account: str + chat_id: int + msg_id: int + received_at: float + due_at: float + expires_at: float | None = None + #: `user`, `chat` (basic group), `megagroup` or `channel` (broadcast). + peer: str = "user" + topic_id: int | None = None + grouped_id: int | None = None + #: A multi-step action's progress; `reply` is `""` then `send`. + phase: str = "" + #: Transient failures so far. + attempts: int = 0 + #: FLOOD_WAITs so far, counted apart so a flood does not use up retries. + floods: int = 0 + presence: str = "leave" + quiet_hours: str | None = None + on_takeover: str = "cancel" + dry_run: bool = False + payload: dict[str, Any] = field(default_factory=dict) + #: Identifies the work, not the attempt: `job|index|chat|msg|extra`. + key: str = "" + last_error: str | None = None + #: Runtime only: `pending` or `running`. Never persisted, so an item that + #: was running when the daemon died is simply pending again. + state: str = field(default="pending", compare=False) + + def to_json(self) -> dict[str, Any]: + body = asdict(self) + body.pop("state", None) + return body + + @classmethod + def from_json(cls, body: dict[str, Any]) -> PendingItem: + known = {f.name for f in fields(cls)} - {"state"} + return cls(**{k: v for k, v in body.items() if k in known}) + + +class PendingStore: + """`accounts//pending.json`.""" + + def __init__(self, path: Path) -> None: + self.path = path + + def load(self) -> tuple[list[PendingItem], list[str]]: + try: + raw = json.loads(self.path.read_text(encoding="utf-8")) + except FileNotFoundError: + return [], [] + except (OSError, json.JSONDecodeError) as exc: + log.warning("cannot read %s, starting with an empty queue: %s", self.path, exc) + return [], [] + if not isinstance(raw, dict): + return [], [] + items: list[PendingItem] = [] + for body in raw.get("items") or []: + if not isinstance(body, dict): + continue + try: + items.append(PendingItem.from_json(body)) + except TypeError as exc: + log.warning("skipping an unreadable pending item in %s: %s", self.path, exc) + recent = [str(key) for key in raw.get("recent") or [] if isinstance(key, str)] + return items[:MAX_PENDING], recent[-RECENT_KEYS:] + + def save(self, items: list[PendingItem], recent: list[str]) -> None: + self.save_raw([item.to_json() for item in items], recent) + + def save_raw(self, items: list[dict[str, Any]], recent: list[str]) -> None: + """Write already-serialised items; safe to run in a worker thread.""" + from tlgr.core.paths import write_private + + body = { + "version": _VERSION, + "items": items[:MAX_PENDING], + "recent": recent[-RECENT_KEYS:], + } + with contextlib.suppress(OSError): + write_private(self.path, json.dumps(body, ensure_ascii=False, separators=(",", ":"))) diff --git a/tlgr/gateway/presence.py b/tlgr/gateway/presence.py new file mode 100644 index 0000000..47ee101 --- /dev/null +++ b/tlgr/gateway/presence.py @@ -0,0 +1,107 @@ +"""Online status around job actions, per account. + +Telethon never calls `account.updateStatus`, so an account driven by tlgr +reads as offline while it reacts and replies, which is exactly the pattern +that makes automation obvious. The modes: + +* `leave` (the default): never touch presence; +* `blip`: online just before the action, offline about five seconds after. + Overlapping blips merge, so a burst does not flap online/offline; +* `session`: online while any action of this account is running or due + within a few seconds, offline about five seconds after the last. + +Presence is one fact per account, so two jobs asking for different modes have +to agree. The rule: the most "online" request wins while its actions run. A +`leave` action never sends anything; it neither turns presence on nor ends an +online stretch a `blip` or `session` action started. Offline is only sent once +no `blip`/`session` action is running and, for `session`, none is due soon. + +If the account-level `[presence] mode` is not `off`, the daemon already owns +the account's status and this manager stays silent. +""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +from collections.abc import Awaitable, Callable +from typing import Any + +log = logging.getLogger("tlgr.gateway.presence") + +__all__ = ["PresenceManager"] + +#: How long the account stays online after its last action. +LINGER_S = 5.0 +#: A `session` stays online if another `session` action is due this soon. +LOOKAHEAD_S = 5.0 + + +class PresenceManager: + def __init__( + self, + run_op: Callable[[str, dict[str, Any]], Awaitable[Any]], + clock: Any, + *, + enabled: bool = True, + session_due_soon: Callable[[float], bool] | None = None, + ) -> None: + self._run_op = run_op + self._clock = clock + self.enabled = enabled + self._session_due_soon = session_due_soon or (lambda _within: False) + self.online = False + self._holds = 0 + self._linger_until = 0.0 + self._offline_task: asyncio.Task[None] | None = None + + async def before(self, mode: str) -> None: + if not self.enabled or mode == "leave": + return + self._holds += 1 + if not self.online: + await self._set(online=True) + + async def after(self, mode: str) -> None: + if not self.enabled or mode == "leave": + return + self._holds = max(0, self._holds - 1) + self._linger_until = max(self._linger_until, self._clock.now() + LINGER_S) + if self._offline_task is None or self._offline_task.done(): + self._offline_task = asyncio.create_task(self._go_offline_later()) + + async def _go_offline_later(self) -> None: + while self.online: + wait = self._linger_until - self._clock.now() + if wait > 0: + await self._clock.sleep(wait) + continue + if self._holds > 0: + return # the running action's `after` starts a new countdown + if self._session_due_soon(LOOKAHEAD_S): + self._linger_until = self._clock.now() + LINGER_S + continue + await self._set(online=False) + if self._holds > 0: + # An action started while the offline request was in flight. + await self._set(online=True) + return + + async def _set(self, *, online: bool) -> None: + self.online = online + try: + await self._run_op("profile.presence.set", {"state": "online" if online else "offline"}) + except asyncio.CancelledError: + raise + except Exception as exc: + log.debug("presence update failed: %s", exc) + + async def stop(self) -> None: + task, self._offline_task = self._offline_task, None + if task is not None and not task.done(): + task.cancel() + with contextlib.suppress(asyncio.CancelledError, Exception): + await task + if self.enabled and self.online: + await self._set(online=False) diff --git a/tlgr/gateway/scheduler.py b/tlgr/gateway/scheduler.py new file mode 100644 index 0000000..74c0053 --- /dev/null +++ b/tlgr/gateway/scheduler.py @@ -0,0 +1,909 @@ +"""The job-action scheduler: one per account, shared by every job on it. + +A job's bus handler must never sleep: handlers run on per-chat worker lanes, +so a handler waiting out a 90 second delay would stall every later event in +that chat. A job therefore only *plans* (filters, percent roll, payloads) and +hands the result here as `PendingItem`s. Everything slow happens on this +scheduler's own tasks: + +* **delay** - each item has an absolute due time, measured from when the + daemon received the event, not from the message date, so a backlog replayed + by catch-up is not "overdue"; +* **pacing** - one worker and one `Pacer` per action kind, so a backlog of + reactions never delays a forward; +* **coalescing** - reads (per chat and topic) and views (per chat) that are + due together go out as one call; +* **react implies read** - see `ensure_read`; +* **quiet hours** - read, view, react and reply are held until the window + ends, then released through the pacer; +* **expiry** - react, view and reply give up 24 hours after the event; +* **manual takeover** - when the account reads the chat or writes in it from + another device, pending items for messages up to that point are dropped + according to their `on_takeover` mode. tlgr's own reads and sends are told + apart by the ids it recorded doing them; +* **retries** - FLOOD_WAIT reschedules after the wait and slows the queue; + a transient failure retries with backoff; anything else is an error; +* **persistence** - every change is written (debounced) to + `accounts//pending.json`, and `resume()` reloads it at boot. + +Time comes from an injected clock, so tests run hours of pacing in +milliseconds. +""" + +from __future__ import annotations + +import asyncio +import contextlib +import heapq +import itertools +import logging +import random +import re +import uuid +from collections import OrderedDict, deque +from collections.abc import Callable, Iterable +from dataclasses import dataclass, field +from datetime import datetime, tzinfo +from typing import Any + +from tlgr.actions import get_builtin +from tlgr.actions.base import Action, Outcome +from tlgr.gateway.knobs import QuietHours +from tlgr.gateway.pacer import DEFAULT_PACING, Pacer, PacingRule, SystemClock +from tlgr.gateway.pending import MAX_PENDING, RECENT_KEYS, PendingItem, PendingStore +from tlgr.gateway.presence import PresenceManager + +log = logging.getLogger("tlgr.gateway.scheduler") + +__all__ = ["KINDS", "AccountPacing", "ActionScheduler"] + +KINDS: tuple[str, ...] = tuple(DEFAULT_PACING) + +#: Transient failures: retried this many times, after these waits (seconds). +RETRY_BACKOFF_S = (5.0, 30.0, 120.0) +#: An action that never expires (forward, read) keeps retrying a transient +#: failure every ten minutes up to this many attempts, so a relay rides out +#: an account that is reconnecting for an hour instead of dropping posts. +PATIENT_RETRIES = 12 +PATIENT_BACKOFF_S = 600.0 +#: FLOOD_WAITs an item survives before it is counted as an error. +MAX_FLOOD_RETRIES = 10 +#: An outgoing message within this long of tlgr's own send is tlgr's echo. +OWN_SEND_WINDOW_S = 10.0 +#: Items released at the end of quiet hours are spread over this long. +QUIET_RELEASE_SPREAD_S = 300.0 +#: The longest a worker sleeps before looking at the wall clock again. +MAX_WAIT_S = 60.0 +#: Writes to `pending.json` are coalesced over this long. +SAVE_DEBOUNCE_S = 1.0 +_ALBUM_MEMORY = 2000 + +COUNTERS = ("done", "skipped", "superseded", "expired", "errors") + + +@dataclass +class AccountPacing: + """The `pacing.` block of `jobs.yaml`, parsed.""" + + rules: dict[str, PacingRule] = field(default_factory=dict) + #: Kind -> seconds, or None for "never". Absent kinds use the default. + expire: dict[str, float | None] = field(default_factory=dict) + + +@dataclass +class _Album: + roll: bool + item_id: str | None = None + has_caption: bool = False + + +class ActionScheduler: + """Delays, paces, persists and executes one account's job actions.""" + + def __init__( + self, + account: str, + runner: Callable[[str, dict[str, Any]], Any], + *, + store: PendingStore | None = None, + clock: Any = None, + rng: random.Random | None = None, + tz: tzinfo | None = None, + presence_enabled: bool = True, + ) -> None: + self.account = account + self.runner = runner + self.store = store + self.clock = clock or SystemClock() + self.rng = rng or random.Random() + self.tz = tz + self.items: dict[str, PendingItem] = {} + self.pacers: dict[str, Pacer] = { + kind: Pacer(rule, rng=self.rng) for kind, rule in DEFAULT_PACING.items() + } + self.expire: dict[str, float | None] = {} + self.stats: dict[tuple[str, int], dict[str, Any]] = {} + self.presence = PresenceManager( + self._run_op, + self.clock, + enabled=presence_enabled, + session_due_soon=self._session_due_soon, + ) + self._heaps: dict[str, list[tuple[float, int, str]]] = {kind: [] for kind in KINDS} + self._wake: dict[str, asyncio.Event] = {kind: asyncio.Event() for kind in KINDS} + self._seq = itertools.count() + self._workers: dict[str, asyncio.Task[None]] = {} + self._running: set[str] = set() + #: Running items whose job was removed or cancelled meanwhile. + self._cancelled: set[str] = set() + self._tasks: set[asyncio.Task[Any]] = set() + self._keys: dict[str, str] = {} + self._recent: OrderedDict[str, None] = OrderedDict() + self._albums: OrderedDict[tuple[Any, ...], _Album] = OrderedDict() + self._quiet: dict[str, QuietHours] = {} + #: `(chat, topic) -> highest id known read`, by tlgr or by the user. + self._read_marks: dict[tuple[int, int | None], int] = {} + #: `(chat, topic) -> highest id tlgr itself asked to read`. + self._own_read: dict[tuple[int, int | None], int] = {} + self._own_sent: dict[int, deque[int]] = {} + self._own_send_at: dict[int, float] = {} + self._save_handle: asyncio.TimerHandle | None = None + self._save_task: asyncio.Future[None] | None = None + self._held: list[PendingItem] = [] + self._loaded = False + self._resumed = False + self._closed = False + self._full_logged = False + + # -- configuration ---------------------------------------------------- + + def configure(self, pacing: AccountPacing | None) -> None: + pacing = pacing or AccountPacing() + for kind, default in DEFAULT_PACING.items(): + self.pacers[kind].configure(pacing.rules.get(kind, default)) + self.expire = dict(pacing.expire) + + def expiry_for(self, action: Action) -> float | None: + if action.name in self.expire: + return self.expire[action.name] + return action.default_expiry + + # -- lifecycle ---------------------------------------------------------- + + def start(self) -> None: + for kind in KINDS: + task = self._workers.get(kind) + if task is None or task.done(): + task = asyncio.create_task( + self._worker(kind), name=f"tlgr-actions:{self.account}:{kind}" + ) + task.add_done_callback(lambda done, kind=kind: self._restart_worker(kind, done)) + self._workers[kind] = task + + def close(self) -> None: + """Start no new action; the one running now may finish.""" + self._closed = True + for event in self._wake.values(): + event.set() + + async def stop(self, *, timeout: float = 5.0) -> None: + """Let a running action finish (briefly), stop, and persist the queue.""" + self.close() + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout + while self._running and loop.time() < deadline: + await asyncio.sleep(0.05) + for task in [*self._workers.values(), *self._tasks]: + task.cancel() + for task in [*self._workers.values(), *self._tasks]: + with contextlib.suppress(asyncio.CancelledError, Exception): + await task + self._workers.clear() + self._tasks.clear() + await self.presence.stop() + if self._save_task is not None: + with contextlib.suppress(Exception): + await self._save_task + self.flush() + + def preload(self) -> None: + """Read the persisted queue now, before any job can submit. + + The finished-item keys take effect at once, so an update replayed by + catch-up in the first seconds after a restart is still recognised. + The items wait in `_held` until `resume` knows which jobs exist. + """ + if self._loaded or self.store is None: + self._loaded = True + return + self._loaded = True + self._held, recent = self.store.load() + for key in recent: + self._recent[key] = None + for item in self._held: + # Held keys count as pending: a replay must not schedule a copy. + if item.key: + self._recent[item.key] = None + + def resume(self, active_jobs: Iterable[str]) -> int: + """Queue the persisted items whose jobs still exist, once.""" + self.preload() + if self._resumed: + return 0 + self._resumed = True + held, self._held = self._held, [] + active = set(active_jobs) + restored = dropped = 0 + for item in held: + if item.job not in active or get_builtin(item.action) is None: + dropped += 1 + continue + if item.id in self.items or (item.key and item.key in self._keys): + continue + if item.key: + self._recent.pop(item.key, None) + self._insert(item) + restored += 1 + if dropped: + log.info( + "dropped %d pending action(s) of jobs that no longer exist on %s", + dropped, + self.account, + ) + if restored or dropped: + self._dirty() + if restored: + log.info("resumed %d pending action(s) on %s", restored, self.account) + return restored + + # -- submission ----------------------------------------------------------- + + def album_roll(self, key: tuple[Any, ...], percent: int) -> bool: + """One percent roll per album, so a ten-photo album is not ten rolls.""" + album = self._albums.get(key) + if album is None: + album = _Album(roll=self.roll(percent)) + self._albums[key] = album + while len(self._albums) > _ALBUM_MEMORY: + self._albums.popitem(last=False) + return album.roll + + def roll(self, percent: int) -> bool: + if percent >= 100: + return True + if percent <= 0: + return False + return self.rng.randrange(100) < percent + + def submit( + self, item: PendingItem, *, album: tuple[Any, ...] | None = None, has_caption: bool = False + ) -> str: + """Queue *item*. Returns `scheduled`, `duplicate`, `album` or `full`.""" + if item.key and (item.key in self._keys or item.key in self._recent): + return "duplicate" + if album is not None: + memo = self._albums.get(album) + if memo is not None and memo.item_id is not None: + existing = self.items.get(memo.item_id) + if ( + existing is not None + and existing.state == "pending" + and has_caption + and not memo.has_caption + ): + # The caption message is where the album's reaction lives. + existing.msg_id = item.msg_id + memo.has_caption = True + self._dirty() + self.count(item.job, item.index, item.action, "skipped") + return "album" + if len(self.items) >= MAX_PENDING: + if not self._full_logged: + log.error( + "the action queue for %s is full (%d items); refusing new ones", + self.account, + MAX_PENDING, + ) + self._full_logged = True + self.count(item.job, item.index, item.action, "errors", error="queue full") + return "full" + self._full_logged = False + if album is not None: + memo = self._albums.setdefault(album, _Album(roll=True)) + memo.item_id = item.id + memo.has_caption = has_caption + self._insert(item) + self._dirty() + return "scheduled" + + def _insert(self, item: PendingItem) -> None: + item.state = "pending" + self.items[item.id] = item + if item.key: + self._keys[item.key] = item.id + self.stats_for(item.job, item.index, item.action) + self._push(item) + + def _push(self, item: PendingItem) -> None: + kind = item.action + heapq.heappush(self._heaps[kind], (item.due_at, next(self._seq), item.id)) + self._wake[kind].set() + + @staticmethod + def new_id() -> str: + return uuid.uuid4().hex[:10] + + # -- counters ------------------------------------------------------------- + + def stats_for(self, job: str, index: int, action: str) -> dict[str, Any]: + key = (job, index) + found = self.stats.get(key) + if found is None or found.get("action") != action: + found = {"action": action, **dict.fromkeys(COUNTERS, 0), "last_error": None} + self.stats[key] = found + return found + + def count( + self, job: str, index: int, action: str, outcome: str, *, error: str | None = None + ) -> None: + stats = self.stats_for(job, index, action) + stats[outcome] = int(stats.get(outcome) or 0) + 1 + if error: + stats["last_error"] = error + + def job_stats(self, job: str) -> list[dict[str, Any]]: + pending: dict[int, int] = {} + for item in self.items.values(): + if item.job == job: + pending[item.index] = pending.get(item.index, 0) + 1 + rows = [] + for (name, index), stats in sorted(self.stats.items(), key=lambda kv: kv[0][1]): + if name != job: + continue + rows.append({"index": index, **stats, "pending": pending.get(index, 0)}) + return rows + + def forget_job(self, job: str) -> int: + """A job was removed: drop its pending items and its counters.""" + dropped = [item for item in self.items.values() if item.job == job] + for item in dropped: + if item.id in self._running: + self._cancelled.add(item.id) + else: + self._remove(item) + for key in [key for key in self.stats if key[0] == job]: + del self.stats[key] + if dropped: + self._dirty() + return len(dropped) + + # -- the queue as data ------------------------------------------------------ + + def is_running(self, item_id: str) -> bool: + return item_id in self._running + + def pending(self) -> list[PendingItem]: + return sorted(self.items.values(), key=lambda item: item.due_at) + + def cancel( + self, + *, + ids: Iterable[str] = (), + chat_id: int | None = None, + job: str | None = None, + everything: bool = False, + ) -> list[PendingItem]: + """Drop matching items that are not running; they count as superseded.""" + wanted = set(ids) + cancelled: list[PendingItem] = [] + for item in list(self.items.values()): + if item.id in self._running: + if everything or (job is not None and item.job == job and not wanted): + self._cancelled.add(item.id) + continue + if not everything: + if wanted and item.id not in wanted: + continue + if chat_id is not None and item.chat_id != chat_id: + continue + if job is not None and item.job != job: + continue + if not wanted and chat_id is None and job is None: + continue + self._finish(item, "superseded") + cancelled.append(item) + return cancelled + + def snapshot(self) -> dict[str, Any]: + by_kind = dict.fromkeys(KINDS, 0) + for item in self.items.values(): + by_kind[item.action] = by_kind.get(item.action, 0) + 1 + return { + "pending": len(self.items), + "by_action": by_kind, + "running": len(self._running), + "presence_online": self.presence.online, + "pacers": {kind: pacer.snapshot() for kind, pacer in self.pacers.items()}, + } + + # -- persistence ------------------------------------------------------------ + + def _dirty(self) -> None: + if self.store is None or self._save_handle is not None: + return + try: + loop = asyncio.get_running_loop() + except RuntimeError: + self.flush() + return + self._save_handle = loop.call_later(SAVE_DEBOUNCE_S, self._save_soon) + + def _save_soon(self) -> None: + """The debounced save: serialise here, write in a thread.""" + self._save_handle = None + if self.store is None: + return + self.preload() + items = [item.to_json() for item in [*self.pending(), *self._held]] + recent = list(self._recent) + store = self.store + previous = self._save_task + + async def write() -> None: + if previous is not None: + with contextlib.suppress(Exception): + await previous + await asyncio.to_thread(store.save_raw, items, recent) + + self._save_task = asyncio.ensure_future(write()) + + def flush(self) -> None: + if self._save_handle is not None: + self._save_handle.cancel() + self._save_handle = None + if self.store is None: + return + # Never overwrite a queue that was not read yet: read it, keep its + # items held, and write them back beside ours. + self.preload() + self.store.save([*self.pending(), *self._held], list(self._recent)) + + # -- the workers -------------------------------------------------------------- + + def _peek(self, kind: str) -> PendingItem | None: + heap = self._heaps[kind] + while heap: + due, _, item_id = heap[0] + item = self.items.get(item_id) + if ( + item is None + or item.state != "pending" + or item.due_at != due + or item.id in self._running + ): + heapq.heappop(heap) + continue + return item + return None + + async def _wait(self, kind: str, timeout: float | None) -> None: + event = self._wake[kind] + if timeout is None: + await event.wait() + return + if timeout <= 0: + return + # Due times are wall-clock, sleeps are monotonic; after the machine + # sleeps they disagree. Re-checking every minute bounds the drift. + sleeper = asyncio.ensure_future(self.clock.sleep(min(timeout, MAX_WAIT_S))) + waiter = asyncio.ensure_future(event.wait()) + try: + await asyncio.wait({sleeper, waiter}, return_when=asyncio.FIRST_COMPLETED) + finally: + sleeper.cancel() + waiter.cancel() + + async def _worker(self, kind: str) -> None: + # A bug in one step must not end the worker: a dead worker would stop + # that kind on this account for good, silently. + while not self._closed: + try: + await self._step(kind) + except asyncio.CancelledError: + if self._cancel_is_ours(): + raise + log.warning("the %s worker for %s was cancelled from outside", kind, self.account) + except Exception: + log.exception("the %s worker for %s hit an unexpected error", kind, self.account) + await self.clock.sleep(1.0) + + def _cancel_is_ours(self) -> bool: + """Was the running task cancelled by `stop()` (or the loop), not a request?""" + if self._closed: + return True + task = asyncio.current_task() + cancelling = getattr(task, "cancelling", None) + return bool(cancelling()) if cancelling is not None else False + + def _restart_worker(self, kind: str, task: asyncio.Task[None]) -> None: + """The backstop: a worker that ended while the scheduler runs is restarted.""" + if self._closed: + return + log.warning("restarting the %s worker for %s", kind, self.account) + self._workers.pop(kind, None) + self.start() + + async def _step(self, kind: str) -> None: + """Wait for, or run, the next item of *kind*.""" + pacer = self.pacers[kind] + self._wake[kind].clear() + item = self._peek(kind) + if item is None: + await self._wait(kind, None) + return + now = self.clock.now() + if item.due_at > now: + await self._wait(kind, item.due_at - now) + return + action = get_builtin(item.action) + if action is None: + self._finish(item, "errors", error=f"unknown action {item.action!r}") + return + if item.expires_at is not None and now >= item.expires_at: + self._finish(item, "expired") + return + if self._hold_for_quiet_hours(item, action, now): + return + if not item.dry_run: + wait = pacer.delay(now) + if wait > 0: + await self._wait(kind, wait) + return + batch = self._batch_for(item, action, now) + if not item.dry_run: + pacer.consume(now) + await self._execute(action, batch) + + def _batch_for(self, item: PendingItem, action: Action, now: float) -> list[PendingItem]: + key = action.batch_key(item) + if key is None: + return [item] + batch = [ + other + for other in self.items.values() + if other.action == item.action + and other.state == "pending" + and other.id not in self._running + and other.due_at <= now + and action.batch_key(other) == key + and (other.expires_at is None or other.expires_at > now) + and self._quiet_until(other, action, now) is None + ] + batch.sort(key=lambda other: other.due_at) + batch = batch[:100] + return batch if any(other is item for other in batch) else [item, *batch[:99]] + + def _quiet_until(self, item: PendingItem, action: Action, now: float) -> float | None: + """When *item*'s quiet hours end, or None if it may run now.""" + if not action.quiet_hold or not item.quiet_hours: + return None + quiet = self._quiet.get(item.quiet_hours) + if quiet is None: + try: + quiet = QuietHours.parse(item.quiet_hours) + except ValueError: + return None + self._quiet[item.quiet_hours] = quiet + local = datetime.fromtimestamp(now, self.tz) + if not quiet.contains(local): + return None + return quiet.window_end(local).timestamp() + + def _hold_for_quiet_hours(self, item: PendingItem, action: Action, now: float) -> bool: + release = self._quiet_until(item, action, now) + if release is None: + return False + self._reschedule(item, release + self.rng.uniform(0.0, QUIET_RELEASE_SPREAD_S)) + return True + + def _reschedule(self, item: PendingItem, due_at: float) -> None: + item.due_at = due_at + item.state = "pending" + self._push(item) + self._dirty() + + async def _execute(self, action: Action, batch: list[PendingItem]) -> None: + for item in batch: + item.state = "running" + self._running.add(item.id) + first = batch[0] + try: + if first.dry_run: + log.info("[%s] dry run: would %s", first.job, action.describe(batch)) + outcome = Outcome() + else: + await self.presence.before(first.presence) + try: + outcome = await action.execute(batch, _Runtime(self)) + finally: + await self.presence.after(first.presence) + except asyncio.CancelledError as cancelled: + if self._cancel_is_ours(): + for item in batch: + item.state = "pending" + raise + # Telethon cancels every request in flight when the client + # disconnects (`daemon reconnect`, a proxy change). That is a + # transient failure of this action, not a request to stop. + self._failed(action, batch, ConnectionError(f"request cancelled: {cancelled}")) + return + except Exception as exc: + self._failed(action, batch, exc) + return + finally: + for item in batch: + self._running.discard(item.id) + for item in batch: + if self._doomed(item): + self._finish(item, "superseded") + elif outcome.status == "later" and outcome.due_at is not None: + self._reschedule(item, outcome.due_at) + else: + self._finish(item, "skipped" if outcome.status == "skipped" else "done") + + def _failed(self, action: Action, batch: list[PendingItem], exc: BaseException) -> None: + from tlgr.core.errors import classify + + body = classify(exc) + now = self.clock.now() + message = f"{body.code}: {_rpc_name(exc)}{body.message}" + if body.code == "RATE_LIMITED": + wait = float(body.wait_seconds or 30) + self.pacers[action.name].penalize(now, wait) + log.warning( + "[%s] %s hit FLOOD_WAIT %ss on %s; rescheduled", + batch[0].job, + action.name, + int(wait), + self.account, + ) + for item in batch: + item.floods += 1 + item.last_error = message + if self._doomed(item): + self._finish(item, "superseded") + elif item.floods > MAX_FLOOD_RETRIES: + self._finish(item, "errors", error=message) + else: + self._reschedule(item, now + wait + self.rng.uniform(1.0, 5.0)) + return + patient = self.expiry_for(action) is None + limit = PATIENT_RETRIES if patient else len(RETRY_BACKOFF_S) + for item in batch: + item.attempts += 1 + item.last_error = message + if self._doomed(item): + self._finish(item, "superseded") + elif body.retryable and item.attempts <= limit: + base = ( + RETRY_BACKOFF_S[item.attempts - 1] + if item.attempts <= len(RETRY_BACKOFF_S) + else PATIENT_BACKOFF_S + ) + backoff = base * self.rng.uniform(1.0, 1.5) + log.info( + "[%s] %s failed (%s); retrying in %ds", + item.job, + action.name, + message, + int(backoff), + ) + self._reschedule(item, now + backoff) + else: + log.warning("[%s] %s failed: %s", item.job, action.name, message) + self._finish(item, "errors", error=message) + + def _finish(self, item: PendingItem, outcome: str, *, error: str | None = None) -> None: + self._remove(item) + if item.key: + self._recent[item.key] = None + while len(self._recent) > RECENT_KEYS: + self._recent.popitem(last=False) + self.count(item.job, item.index, item.action, outcome, error=error) + self._dirty() + + def _doomed(self, item: PendingItem) -> bool: + """Cancelled while it was running: finish it, never requeue it.""" + return item.id in self._cancelled + + def _remove(self, item: PendingItem) -> None: + self._cancelled.discard(item.id) + self.items.pop(item.id, None) + if item.key and self._keys.get(item.key) == item.id: + del self._keys[item.key] + + def _session_due_soon(self, within: float) -> bool: + horizon = self.clock.now() + within + return any( + item.presence == "session" and not item.dry_run and item.due_at <= horizon + for item in self.items.values() + ) + + async def _run_op(self, op: str, request: dict[str, Any]) -> Any: + result = self.runner(op, request) + if asyncio.iscoroutine(result) or isinstance(result, asyncio.Future): + return await result + return result + + def _spawn(self, coro: Any, what: str) -> None: + async def guarded() -> None: + try: + await coro + except asyncio.CancelledError: + raise + except Exception as exc: + log.debug("%s failed: %s", what, exc) + + task = asyncio.create_task(guarded()) + self._tasks.add(task) + task.add_done_callback(self._tasks.discard) + + # -- reads, sends and takeover ---------------------------------------------------- + + def read_mark(self, chat_id: int, topic_id: int | None) -> int: + return self._read_marks.get((chat_id, topic_id), 0) + + def begin_read(self, chat_id: int, topic_id: int | None, max_id: int) -> None: + key = (chat_id, topic_id) + # Before the RPC: the server's echo can arrive before the answer does. + self._own_read[key] = max(self._own_read.get(key, 0), max_id) + + def note_read(self, chat_id: int, topic_id: int | None, max_id: int) -> None: + key = (chat_id, topic_id) + self._read_marks[key] = max(self._read_marks.get(key, 0), max_id) + + def begin_send(self, chat_id: int) -> None: + self._own_send_at[chat_id] = self.clock.now() + + def note_sent(self, chat_id: int, ids: list[int]) -> None: + if not chat_id: + return + sent = self._own_sent.setdefault(chat_id, deque(maxlen=200)) + sent.extend(i for i in ids if i) + self._own_send_at[chat_id] = self.clock.now() + + async def ensure_read(self, item: PendingItem) -> None: + """Read the chat up to *item* before acting on it (react implies read). + + Paced on the read queue, and coalesced: reads already pending for the + chat at or below this message are satisfied by this one and finish. + """ + key = (item.chat_id, item.topic_id) + target = item.msg_id + if self._read_marks.get(key, 0) < target: + pacer = self.pacers["read"] + while (wait := pacer.delay(self.clock.now())) > 0: + await self.clock.sleep(wait) + pacer.consume(self.clock.now()) + request: dict[str, Any] = {"chat": str(item.chat_id), "up_to": target} + if item.topic_id: + request["topic"] = item.topic_id + self.begin_read(item.chat_id, item.topic_id, target) + await self._run_op("message.read", request) + self.note_read(item.chat_id, item.topic_id, target) + for other in list(self.items.values()): + if ( + other.action == "read" + and not other.dry_run + and other.id not in self._running + and (other.chat_id, other.topic_id) == key + and other.msg_id <= target + and not other.payload.get("mentions") + and not other.payload.get("reactions") + ): + self._finish(other, "done") + + async def on_bus(self, envelope: Any, raw: Any) -> None: + """Watch for the account acting in a chat from another device.""" + if raw is None or getattr(envelope, "account", None) != self.account: + return + name = type(raw).__name__ + if name == "UpdateReadHistoryInbox": + from telethon import utils + + chat = int(utils.get_peer_id(raw.peer)) + self.external_read(chat, getattr(raw, "top_msg_id", None), int(raw.max_id)) + elif name == "UpdateReadChannelInbox": + self.external_read(-1000000000000 - int(raw.channel_id), None, int(raw.max_id)) + elif name == "UpdateReadChannelDiscussionInbox": + self.external_read( + -1000000000000 - int(raw.channel_id), int(raw.top_msg_id), int(raw.read_max_id) + ) + elif name in ("UpdateNewMessage", "UpdateNewChannelMessage"): + message = raw.message + if type(message).__name__ == "Message" and getattr(message, "out", False): + from telethon import utils + + self.outgoing(int(utils.get_peer_id(message.peer_id)), int(message.id)) + elif name == "UpdateShortMessage" and getattr(raw, "out", False): + self.outgoing(int(raw.user_id), int(raw.id)) + elif name == "UpdateShortChatMessage" and getattr(raw, "out", False): + self.outgoing(-int(raw.chat_id), int(raw.id)) + + def external_read(self, chat_id: int, topic_id: int | None, max_id: int) -> None: + key = (chat_id, topic_id) + self._read_marks[key] = max(self._read_marks.get(key, 0), max_id) + if max_id <= self._own_read.get(key, 0): + return # the echo of a read tlgr sent + sent_at = self._own_send_at.get(chat_id) + if sent_at is not None and self.clock.now() - sent_at < OWN_SEND_WINDOW_S: + return # sending marks the chat read; that read is tlgr's too + self._takeover(chat_id, max_id, "read elsewhere", topic_id=topic_id) + + def outgoing(self, chat_id: int, msg_id: int) -> None: + if msg_id in self._own_sent.get(chat_id, ()): + return + sent_at = self._own_send_at.get(chat_id) + if sent_at is not None and self.clock.now() - sent_at < OWN_SEND_WINDOW_S: + return + self._takeover(chat_id, msg_id, "sent from another device") + + def _takeover(self, chat_id: int, up_to: int, why: str, *, topic_id: int | None = None) -> None: + dropped = 0 + for item in list(self.items.values()): + if item.chat_id != chat_id or item.msg_id > up_to or item.id in self._running: + continue + if topic_id is not None and item.topic_id != topic_id: + continue # forum ids span the chat; a read in one topic says nothing of another + action = get_builtin(item.action) + if action is None or item.on_takeover not in action.cancelled_by: + continue + self._finish(item, "superseded") + dropped += 1 + if dropped: + log.info( + "manual takeover in %s (%s): dropped %d pending action(s)", chat_id, why, dropped + ) + + +def _rpc_name(exc: BaseException) -> str: + """`ReactionInvalidError` -> `"REACTION_INVALID: "`, the name the docs use.""" + name = type(exc).__name__ + if getattr(exc, "code", None) is None or not name.endswith("Error") or name == "RPCError": + return "" + return re.sub(r"(? None: + self._s = scheduler + self.rng = scheduler.rng + + def now(self) -> float: + return float(self._s.clock.now()) + + async def op(self, op: str, request: dict[str, Any]) -> Any: + return await self._s._run_op(op, request) + + async def ensure_read(self, item: Any) -> None: + await self._s.ensure_read(item) + + def read_mark(self, chat_id: int, topic_id: int | None) -> int: + return self._s.read_mark(chat_id, topic_id) + + def begin_read(self, chat_id: int, topic_id: int | None, max_id: int) -> None: + self._s.begin_read(chat_id, topic_id, max_id) + + def note_read(self, chat_id: int, topic_id: int | None, max_id: int) -> None: + self._s.note_read(chat_id, topic_id, max_id) + + def begin_send(self, chat_id: int) -> None: + self._s.begin_send(chat_id) + + def note_sent(self, chat_id: int, ids: list[int]) -> None: + self._s.note_sent(chat_id, ids) + + def spawn(self, coro: Any, *, what: str) -> None: + self._s._spawn(coro, what) diff --git a/tlgr/models/daemon.py b/tlgr/models/daemon.py index 20564d5..ad7cc83 100644 --- a/tlgr/models/daemon.py +++ b/tlgr/models/daemon.py @@ -107,6 +107,8 @@ class DaemonStatus(Model, omit_defaults=False): events: EventBusStatus | None = None webhook: dict[str, Any] = {} jobs: list[dict[str, Any]] = [] + #: Per account: the job-action queue (pending by action, pacers, presence). + actions: dict[str, Any] = {} # v1's `/daemon/status` carried these two, and AGENT.md documents them. connections: dict[str, bool] = {} disconnected: list[str] = [] @@ -250,12 +252,37 @@ class DeadLetterResult(Model): dry_run: bool = False +class ActionCounters(Model, omit_defaults=False): + """What one action of a job has done since the daemon started. + + `skipped` is a percent roll that said no, or a message with nothing to do + (no voice note to listen to, an album sibling); `superseded` is a pending + action dropped because the account acted in the chat from another device + or an operator cancelled it; `expired` is one that waited past its expiry. + """ + + index: int = 0 + action: str = "" + done: int = 0 + skipped: int = 0 + superseded: int = 0 + expired: int = 0 + pending: int = 0 + errors: int = 0 + last_error: str | None = None + + class JobState(Model, omit_defaults=False): """One gateway job. `omit_defaults=False` because `enabled` and `running` are the answer: with the default, an enabled job dropped `enabled` and a stopped one dropped `running`, and `job list` printed `-` for both. + + `skipped` is the job-level filter count (events the job's filters + rejected); the per-action counters are in `action_counters`, and `done`, + `superseded`, `expired` and `pending` are their totals. `errors` is the + job's own errors plus every action's. """ name: str @@ -272,6 +299,39 @@ class JobState(Model, omit_defaults=False): errors: int = 0 last_match_at: str | None = None last_error: str | None = None + done: int = 0 + superseded: int = 0 + expired: int = 0 + pending: int = 0 + action_counters: list[ActionCounters] = [] + + +class PendingAction(Model, omit_defaults=False): + """One job action waiting for its delay, its pacer slot or quiet hours.""" + + id: str + account: str = "" + job: str = "" + action: str = "" + chat_id: int = 0 + msg_id: int = 0 + #: `waiting` (before its due time), `due` (waiting for a pacer slot) or + #: `running`. + state: str = "waiting" + due_at: str | None = None + eta_s: int = 0 + expires_at: str | None = None + attempts: int = 0 + dry_run: bool = False + detail: str = "" + last_error: str | None = None + + +class QueueCancel(Model, omit_defaults=False): + """`job queue cancel`: how many pending actions were dropped, and which.""" + + cancelled: int = 0 + ids: list[str] = [] class Job(Model): diff --git a/tlgr/ops/config.py b/tlgr/ops/config.py index 7dda924..322eeb6 100644 --- a/tlgr/ops/config.py +++ b/tlgr/ops/config.py @@ -1077,13 +1077,17 @@ async def config_validate(ctx: OpContext, req: ConfigValidateReq) -> ValidationR def _validate_jobs(base: Any) -> list[ValidationIssue]: from tlgr.actions import get_action - from tlgr.gateway.config import load_gateway_configs + from tlgr.gateway.config import load_jobs_file issues: list[ValidationIssue] = [] try: - configs = load_gateway_configs(base) + jobs_file = load_jobs_file(base) except Exception as exc: return [ValidationIssue(file="jobs.yaml", message=str(exc))] + # Unknown keys, bad durations, percents, presence modes, pacing: the same + # parser the daemon runs, so "validate passed" means "the daemon loads it". + issues.extend(ValidationIssue(file="jobs.yaml", message=p) for p in jobs_file.problems) + configs = jobs_file.jobs for config in configs: if not config.name: issues.append(ValidationIssue(file="jobs.yaml", message="a job has no `name`")) diff --git a/tlgr/ops/daemon.py b/tlgr/ops/daemon.py index 59436a7..32d5bdc 100644 --- a/tlgr/ops/daemon.py +++ b/tlgr/ops/daemon.py @@ -775,6 +775,7 @@ async def daemon_status(ctx: OpContext, req: DaemonStatusReq) -> DaemonStatus: else None, webhook=status.get("webhook") or {}, jobs=status.get("jobs") or [], + actions=status.get("actions") or {}, connections={row.alias: row.state == "online" for row in rows}, disconnected=sorted(row.alias for row in rows if row.state != "online"), ) diff --git a/tlgr/ops/job.py b/tlgr/ops/job.py index bdbde83..6e8389f 100644 --- a/tlgr/ops/job.py +++ b/tlgr/ops/job.py @@ -25,7 +25,14 @@ from tlgr.core.errors import EXIT_EMPTY, NotFoundError, UsageError from tlgr.core.pagination import PageKind, build_page from tlgr.models.base import Request -from tlgr.models.daemon import Job, JobState, JobTestFrame +from tlgr.models.daemon import ( + ActionCounters, + Job, + JobState, + JobTestFrame, + PendingAction, + QueueCancel, +) from tlgr.models.page import Page from tlgr.models.peer import PeerRef from tlgr.ops._params import arg, opt @@ -120,18 +127,15 @@ async def job_list(ctx: OpContext, req: JobListReq) -> Page[JobState]: account = str(entry.get("account", "")) if ctx.account and ctx.account != "all" and account and account != ctx.account: continue - rows.append( - JobState( - name=name, - account=account, - enabled=enabled, - running=bool(running.get("running")), - events=[str(e) for e in (entry.get("events") or ["new_message"])], - matched=int(running.get("matched") or 0), - skipped=int(running.get("skipped") or 0), - errors=int(running.get("errors") or 0), - ) + state = JobState( + name=name, + account=account, + enabled=enabled, + running=bool(running.get("running")), + events=[str(e) for e in (entry.get("events") or ["new_message"])], ) + _apply_counters(state, running) + rows.append(state) return build_page( rows, op="job.list", @@ -141,6 +145,40 @@ async def job_list(ctx: OpContext, req: JobListReq) -> Page[JobState]: ) +def _apply_counters(state: JobState, running: dict[str, Any]) -> None: + """Copy the engine's counters onto *state*: the job's own and each action's.""" + state.matched = int(running.get("matched") or 0) + state.skipped = int(running.get("skipped") or 0) + errors = int(running.get("errors") or 0) + counters: list[ActionCounters] = [] + for row in running.get("actions") or []: + if not isinstance(row, dict): + continue + counters.append( + ActionCounters( + index=int(row.get("index") or 0), + action=str(row.get("action") or ""), + done=int(row.get("done") or 0), + skipped=int(row.get("skipped") or 0), + superseded=int(row.get("superseded") or 0), + expired=int(row.get("expired") or 0), + pending=int(row.get("pending") or 0), + errors=int(row.get("errors") or 0), + last_error=row.get("last_error"), + ) + ) + state.action_counters = counters + state.done = sum(c.done for c in counters) + state.superseded = sum(c.superseded for c in counters) + state.expired = sum(c.expired for c in counters) + state.pending = sum(c.pending for c in counters) + state.actions_run = state.done + state.errors = errors + sum(c.errors for c in counters) + last = [c.last_error for c in counters if c.last_error] + if last: + state.last_error = last[-1] + + SPEC_JOB_LIST = OperationSpec( id="job.list", request=JobListReq, @@ -159,7 +197,7 @@ async def job_list(ctx: OpContext, req: JobListReq) -> Page[JobState]: idempotent=True, rate_class="local", timeout_s=30, - columns=("name", "account", "enabled", "running", "matched", "errors"), + columns=("name", "account", "enabled", "running", "matched", "pending", "done", "errors"), example={ "items": [ { @@ -168,6 +206,10 @@ async def job_list(ctx: OpContext, req: JobListReq) -> Page[JobState]: "enabled": True, "running": True, "events": ["new_message"], + "matched": 12, + "done": 12, + "pending": 1, + "action_counters": [{"index": 0, "action": "forward", "done": 12, "pending": 1}], } ], "has_more": False, @@ -187,7 +229,7 @@ class JobGetReq(Request): async def job_get(ctx: OpContext, req: JobGetReq) -> JobState: - """One job's resolved pipeline: filters, processors, actions.""" + """One job's resolved pipeline (filters, processors, actions) and its counters.""" entry = _find(_load_raw(), req.name) state = JobState( name=req.name, @@ -198,6 +240,13 @@ async def job_get(ctx: OpContext, req: JobGetReq) -> JobState: processors=[str(p) for p in (entry.get("processors") or [])], actions=[a for a in (entry.get("actions") or []) if isinstance(a, dict)], ) + daemon = getattr(ctx, "daemon", None) + if daemon is not None: + running: dict[str, Any] = next( + (row for row in daemon.list_jobs() if row.get("name") == req.name), {} + ) + state.running = bool(running.get("running")) + _apply_counters(state, running) if req.explain: state.filters = { key: {"value": value, "resolves_to": _explain_filter(key)} @@ -226,11 +275,10 @@ def _explain_filter(name: str) -> str: request=JobGetReq, response=JobState, impl=job_get, - summary="Show one job's resolved pipeline (filters, processors, actions)", + summary="Show one job's resolved pipeline (filters, processors, actions) and counters", needs_account=False, - needs_auth=False, needs_client=False, - surface=Surface.LOCAL, + surface=Surface.DAEMON, idempotent=True, rate_class="local", timeout_s=15, @@ -279,7 +327,7 @@ def _coerce(raw: str) -> Any: def _action(spec: str) -> dict[str, Any]: - """`reply:hello` or `forward:to=@archive` → one action entry.""" + """`reply:hello`, `forward:to=@archive`, `react:emoji=👍`, `read:delay=10-90s`.""" name, sep, rest = spec.partition(":") name = name.strip() if not name: @@ -313,6 +361,14 @@ class JobAddReq(Request): processor: Annotated[ list[str], opt("--processor", metavar="NAME", help="Processor entry (repeatable).") ] = [] + knob: Annotated[ + list[str], + opt( + "--knob", + metavar="KEY=VALUE", + help="Job-level action default: delay, percent, presence, on_takeover, dry_run.", + ), + ] = [] enabled: Annotated[bool, opt("--enabled/--disabled", help="Initial state.")] = True edit: Annotated[ bool, opt("--edit", help="Open jobs.yaml in $EDITOR instead (the v1 behaviour).") @@ -340,6 +396,7 @@ async def job_add(ctx: OpContext, req: JobAddReq) -> Job: if any(isinstance(e, dict) and e.get("name") == name for e in document["jobs"]): raise UsageError(f"a job named {name!r} already exists; remove it first", field="name") eventtypes.resolve_selectors(entry.get("events") or ["new_message"]) + _validate(entry) document["jobs"].append(entry) added.append(name) @@ -354,6 +411,16 @@ async def job_add(ctx: OpContext, req: JobAddReq) -> Job: ) +def _validate(entry: dict[str, Any]) -> None: + """Refuse a job the engine would refuse, naming every problem in it.""" + from tlgr.gateway.config import JobConfigError, _parse_job + + try: + _parse_job(entry) + except JobConfigError as exc: + raise UsageError("; ".join(exc.problems), field="action") from None + + def _entries_from(req: JobAddReq) -> list[dict[str, Any]]: if req.from_file: return _entries_from_file(req.from_file) @@ -368,6 +435,8 @@ def _entries_from(req: JobAddReq) -> list[dict[str, Any]]: entry["filters"] = _pairs(req.filter, "filter") if req.processor: entry["processors"] = list(req.processor) + if req.knob: + entry.update(_pairs(req.knob, "knob")) if req.action: entry["actions"] = [_action(spec) for spec in req.action] if not entry.get("actions"): @@ -571,25 +640,21 @@ async def job_reload(ctx: OpContext, req: JobReloadReq) -> Job: to run before a reload rather than after one: a config with a typo would otherwise take effect as "that job is gone". """ - from tlgr.gateway.config import load_gateway_configs + from tlgr.gateway.config import load_jobs_file daemon = _runner(ctx) base = getattr(getattr(daemon, "paths", None), "base", None) - configs = load_gateway_configs(base) - problems: list[str] = [] + jobs_file = load_jobs_file(base) + configs = jobs_file.jobs + problems: list[str] = list(jobs_file.problems) for config in configs: - if not config.name: - problems.append("a job has no `name`") if not config.actions: problems.append(f"job {config.name!r} has no actions and would do nothing") - for action in config.actions: - from tlgr.actions import get_action - - if get_action(action.name) is None: - problems.append(f"job {config.name!r} uses unknown action {action.name!r}") if req.validate_only or problems: - return Job(name="", enabled=True, loaded=len(configs), errors=problems) + return Job( + name="", enabled=True, loaded=len(configs) + len(jobs_file.rejected), errors=problems + ) result = await daemon.reload_jobs() return Job( @@ -624,6 +689,216 @@ async def job_reload(ctx: OpContext, req: JobReloadReq) -> Job: ) +# --------------------------------------------------------------------------- +# job queue list / cancel +# --------------------------------------------------------------------------- + + +def _stamp(value: float | None) -> str | None: + if value is None: + return None + from datetime import datetime, timezone + + return datetime.fromtimestamp(value, timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def _detail(item: Any) -> str: + payload = item.payload or {} + if item.action == "react": + return str(payload.get("emoji", "")) + if item.action == "forward": + return f"to {payload.get('to', '')}" + (" (re-send)" if payload.get("resend") else "") + if item.action == "reply": + text = str(payload.get("text", "")) + return (text[:40] + "...") if len(text) > 40 else text + if item.action == "view": + return str(payload.get("mode", "")) + if item.action == "read": + extras = [flag for flag in ("mentions", "reactions") if payload.get(flag)] + return "+".join(extras) + return "" + + +async def _chat_ids(ctx: OpContext, daemon: Any, ref: str | None) -> dict[str, int | None]: + """`--chat` per account: a marked id is itself, `@name` is resolved per account.""" + if ref is None: + return {} + out: dict[str, int | None] = {} + with contextlib.suppress(ValueError): + marked = int(ref) + return dict.fromkeys(daemon.schedulers, marked) + for alias in daemon.schedulers: + client = daemon.get_client(alias) + try: + out[alias] = int(await client.resolve_chat(ref)) if client is not None else None + except Exception: + out[alias] = None + return out + + +def _accounts(ctx: OpContext, daemon: Any) -> dict[str, Any]: + schedulers = dict(daemon.schedulers) + if ctx.account and ctx.account != "all": + return {alias: s for alias, s in schedulers.items() if alias == ctx.account} + return schedulers + + +class JobQueueListReq(Request): + job: Annotated[str | None, opt("--job", metavar="NAME", help="Only this job.")] = None + chat: Annotated[ + str | None, opt("--chat", metavar="CHAT", help="Only this chat (@name or marked id).") + ] = None + action: Annotated[ + str | None, opt("--action", metavar="NAME", help="Only this action (react, read, ...).") + ] = None + + +async def job_queue_list(ctx: OpContext, req: JobQueueListReq) -> Page[PendingAction]: + """Every pending job action: what it is, where, and when it is due.""" + daemon = _runner(ctx) + chats = await _chat_ids(ctx, daemon, req.chat) + rows: list[PendingAction] = [] + for alias, scheduler in sorted(_accounts(ctx, daemon).items()): + now = scheduler.clock.now() + for item in scheduler.pending(): + if req.job and item.job != req.job: + continue + if req.action and item.action != req.action: + continue + if req.chat is not None and item.chat_id != chats.get(alias): + continue + running = scheduler.is_running(item.id) + rows.append( + PendingAction( + id=item.id, + account=alias, + job=item.job, + action=item.action, + chat_id=item.chat_id, + msg_id=item.msg_id, + state="running" if running else "due" if item.due_at <= now else "waiting", + due_at=_stamp(item.due_at), + eta_s=max(0, round(item.due_at - now)), + expires_at=_stamp(item.expires_at), + attempts=item.attempts, + dry_run=item.dry_run, + detail=_detail(item), + last_error=item.last_error, + ) + ) + return build_page( + rows, op="job.queue.list", kind=PageKind.LOCAL, has_more=False, total=len(rows) + ) + + +SPEC_JOB_QUEUE_LIST = OperationSpec( + id="job.queue.list", + request=JobQueueListReq, + response=Page[PendingAction], + impl=job_queue_list, + summary="List pending job actions (delayed, paced or held by quiet hours)", + description=( + "Every action a job has scheduled and not yet run, across accounts, " + "soonest first. `state` is `waiting` before the due time, `due` while " + "it waits for its pacer slot, and `running` while it talks to Telegram." + ), + paginated=PageKind.LOCAL, + needs_account=False, + needs_client=False, + surface=Surface.DAEMON, + idempotent=True, + rate_class="local", + timeout_s=30, + columns=("id", "job", "action", "chat_id", "msg_id", "state", "eta_s", "attempts"), + example={ + "items": [ + { + "id": "3f2a9c1b7e", + "account": "work", + "job": "dm-ack", + "action": "react", + "chat_id": 777123, + "msg_id": 4410, + "state": "waiting", + "due_at": "2026-10-04T09:14:07Z", + "eta_s": 42, + "attempts": 0, + "detail": "👍", + } + ], + "has_more": False, + }, + example_args="job queue list --job dm-ack", + covers_partial=("updates.stream-event-filtering",), + coverage_note="shows what the rules scheduled; the filtering itself is the gateway's.", + tags=frozenset({"agent-safe", "group-default"}), +) + + +class JobQueueCancelReq(Request): + ids: Annotated[ + list[str], + arg(0, metavar="ID", required=False, variadic=True, help="Pending action id(s)."), + ] = [] + chat: Annotated[ + str | None, opt("--chat", metavar="CHAT", help="Everything pending in this chat.") + ] = None + job: Annotated[str | None, opt("--job", metavar="NAME", help="Everything this job has.")] = None + every: Annotated[bool, opt("--all", help="Every pending action on every account.")] = False + + +async def job_queue_cancel(ctx: OpContext, req: JobQueueCancelReq) -> QueueCancel: + """Drop pending job actions by id, chat, job, or all of them. + + A cancelled action counts as `superseded` on its job. An action already + talking to Telegram is left to finish. + """ + if not (req.ids or req.chat or req.job or req.every): + raise UsageError("say what to cancel: an ID, --chat, --job or --all", field="ids") + daemon = _runner(ctx) + chats = await _chat_ids(ctx, daemon, req.chat) + cancelled: list[str] = [] + for alias, scheduler in sorted(_accounts(ctx, daemon).items()): + if req.chat is not None and chats.get(alias) is None: + continue + dropped = scheduler.cancel( + ids=req.ids, + chat_id=chats.get(alias) if req.chat is not None else None, + job=req.job, + everything=req.every, + ) + cancelled.extend(item.id for item in dropped) + if not cancelled: + ctx.mark_already() + return QueueCancel(cancelled=len(cancelled), ids=cancelled) + + +SPEC_JOB_QUEUE_CANCEL = OperationSpec( + id="job.queue.cancel", + request=JobQueueCancelReq, + response=QueueCancel, + impl=job_queue_cancel, + summary="Cancel pending job actions by id, chat, job, or all", + description=( + "Selectors combine: `--job dm-ack --chat @alice` drops only that job's " + "actions in that chat. A cancelled action counts as `superseded`." + ), + mutating=True, + destructive=True, + needs_account=False, + needs_client=False, + surface=Surface.DAEMON, + rate_class="local", + timeout_s=30, + columns=("cancelled",), + example={"cancelled": 2, "ids": ["3f2a9c1b7e", "a81d03c2f4"]}, + example_args="job queue cancel --job dm-ack --yes", + covers_partial=("updates.stream-event-filtering",), + coverage_note="cancels what the rules scheduled; the filtering itself is the gateway's.", + tags=frozenset({"agent-safe"}), +) + + # --------------------------------------------------------------------------- # job test # --------------------------------------------------------------------------- diff --git a/tlgr/registry.py b/tlgr/registry.py index 12db8c4..b4f22cf 100644 --- a/tlgr/registry.py +++ b/tlgr/registry.py @@ -209,6 +209,10 @@ # §12.4 promises stays invocable. "switch", "completion", + # `job queue cancel` drops actions a job scheduled and has not run; + # it is not `delete` (nothing on Telegram is deleted) and not `stop` + # (nothing is running). + "cancel", # PR-12. `profile update` is a path v1 documented, and §12.4 makes a # documented path permanent; `profile set` is its STYLE-shaped alias, # so both spellings reach the one operation.