Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ NG_LLM_REQUEST_TIMEOUT=45s # Classification request deadline
NG_SPAM_LOG_CHANNEL_USERNAME= # Channel for spam logging
NG_SPAM_DEBUG_USER_ID=0 # User allowed to run diagnostics in private chat
NG_SPAM_VERBOSE=false # Verbose in-chat notifications
NG_SPAM_MESSAGE_PROBATION_DURATION=3h # Minimum new-user message probation before a safe exit
NG_SPAM_SAFE_MESSAGES_REQUIRED=3 # Distinct safe new messages needed for author trust
NG_SPAM_AUTHOR_TRUST_DURATION=720h # Trust lifetime; one safe new message renews expired trust
NG_SPAM_MESSAGE_PROBATION_DURATION= # Deprecated; accepted with a warning and ignored
NG_SPAM_VOTING_TIMEOUT=5m # Voting time limit
NG_SPAM_MIN_VOTERS=2 # Minimum required voters
NG_SPAM_MAX_VOTERS=10 # Maximum voters cap
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
- name: Shuffle test
run: go test -shuffle=on ./...
- name: Lint
run: go tool golangci-lint run --enable=unused --enable=unparam --enable=ineffassign --enable=goconst ./...
run: go tool golangci-lint run --no-config --enable=unused --enable=unparam --enable=ineffassign --enable=goconst ./...
- name: Vulnerability scan
run: go tool govulncheck ./...
- name: Validate deployment configuration
Expand Down
10 changes: 6 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ This document serves as the **single source of truth** for all development rules

**ngbot** is a Telegram gatekeeper bot with CAPTCHA verification, LLM-powered spam detection, and community voting moderation.

**Stack**: Go 1.25, SQLite, Telegram Bot API, OpenAI/Gemini LLMs
**Stack**: Go 1.26.8, SQLite, Telegram Bot API, OpenAI/Gemini LLMs

**Structure**:
- `cmd/ngbot/` - Entry point, runtime wiring
Expand Down Expand Up @@ -96,7 +96,9 @@ For detailed architecture, see [docs/CODEBASE_MAP.md](docs/CODEBASE_MAP.md).
- **Join-Captcha WebApp** 🔒: The gatekeeper join-captcha WebApp server speaks plain HTTP and **MUST** run behind a TLS-terminating reverse proxy. Its listen address is configured via `GatekeeperWebApp.ListenAddr`. In the Docker deployment it binds `0.0.0.0:8080` inside the container (mapped to `127.0.0.1:18080` on the host); the default **must NOT** be changed to loopback or the container port mapping breaks.
- **No-Rights Mode** 🛡️: Before banlist, LLM, reaction, or voting moderation, check the bot's restrict-member capability. A confirmed Telegram privilege error is terminal and must not be retried. Public CAPTCHA may still run without restricting the user: success deletes the CAPTCHA; failure leaves a durable 30-minute notice.
- **Manual Allowlist Priority** ✅: A matching per-chat `chat_not_spammer_overrides` row is the only exemption checked before cached or provider-backed banlist enforcement. For every non-allowlisted user, effective banlist membership remains an unconditional deny decision before remembered membership, admin status, commands, voting, or LLM. The always-on cached guard precedes the configured `admin → gatekeeper → reactor` chain, and join updates are checked even when Gatekeeper is disabled. Enforcement directly bans and deletes the available join/current-message artifact without creating a spam case. The guard may inspect the in-memory effective set to short-circuit routing, but it must verify moderation capability before Telegram I/O and must not perform online provider checks in no-rights mode. Allowlist lookup failures fail closed and continue normal moderation.
- **Durable Message Probation** ✏️: A previously untrusted author gets a per-chat durable probation on the first message. Check every new text, caption, or visible `RichMessage` text for the configured minimum duration, then require a distinct safe new message to graduate; commands and empty media start the clock but cannot graduate it. Persist every safe message as `{chat_id, message_id, user_id}` for later edit checks. While probation is active, every authored edit with semantic text is rechecked even without an existing binding, never extends the deadline, and never graduates the author; after graduation, bound-message edits remain protected. Reactions and `chat_known_non_members` never grant message trust. Telegram Bot API has no deletion update for ordinary group messages (`deleted_business_messages` is business-only), so never claim immediate deletion detection. The cached banlist guard applies to all authored edits unless the author has a matching manual allowlist override.
- **Durable Author Trust** ✏️: All chats use a typed `MessageAuthor` (`user` or `sender_chat`); `SenderChat` takes precedence over technical `From`. Three distinct safe new semantic messages grant 30 days of per-chat trust by default; one safe new message renews expired trust. SQLite atomically binds checked messages, increments the counter and grants trust. Membership bookkeeping is separate and best effort after persistence; reactions and `chat_known_non_members` never grant trust. Commands, bot mentions, empty media, reactions and edits never advance or renew admission. Before admission and after expiry all semantic edits are checked; during trust previously bound edits remain protected. Checked-message bindings are durable identity metadata: never expire them with context or trust, because renewal would otherwise make old checked edits unprotected; chat deletion may cascade them. Pending/resolving cases suspend trust; confirmed spam resets it, false positives retain any previous unexpired grant. Channels follow community voting without a mute or technical-user fallback. Keep user allowlist-before-banlist priority; user banlists never consume channel IDs. Preserve linked-channel and anonymous group-admin exemptions.
- **Bounded Conversation Context** 💬: Persist message context separately from author trust in SQLite. Retain original send time, thread/reply links, edited text and Telegram update IDs for same-second edit ordering; snapshots cannot supersede newer authoritative edits. Use only same-chat/thread history younger than 24 hours, at most five earlier replies, 2,000 characters per saved text and 8,000 extra context characters. Prioritize direct reply/quote, source post, then nearby replies. Unknown discussion threads use reply chains, ordinary groups may use recent group history. Successful bot deletions leave short-lived tombstones. Successful user bans that revoke messages clear the matching user context; channel bans do not imply removal of previous channel messages. CAPTCHA context cleanup uses a distinct durable phase so a SQLite retry cannot repeat a completed ban. Ordinary user deletions are not reliably observable through Bot API. Pass history as untrusted structured evidence and classify only the candidate. Never log message texts. No external-reply automatic verdict.
- **Author Trust Configuration** ⚙️: Global `NG_SPAM_SAFE_MESSAGES_REQUIRED` and `NG_SPAM_AUTHOR_TRUST_DURATION` default to `3` and `720h`. Keep the existing per-chat LLM switch. `NG_SPAM_MESSAGE_PROBATION_DURATION` is accepted as deprecated with a warning and has no effect. Migrate prior effective user trust for 30 days from migration, including suspended grants; active probations restart at zero and reaction-only records are excluded. Preserve cases, votes and durable actions.

### Admin Panel UX Rules
- **Cascading Menus** 🧭: Admin settings must be structured as cascading category menus. Do not place many unrelated controls on a single page.
Expand All @@ -114,7 +116,7 @@ For detailed architecture, see [docs/CODEBASE_MAP.md](docs/CODEBASE_MAP.md).
- **Architecture first** 🏛️: Audit before coding: scan repo, read related packages, plan all changes.

### Go Version & Documentation
- **Go Version** 🔢: 1.25 (Latest features where applicable). Ref: [Go Release Notes](https://go.dev/doc/devel/release)
- **Go Version** 🔢: 1.26.8, as pinned in `go.mod` and `Dockerfile`. Ref: [Go Release Notes](https://go.dev/doc/devel/release)
- **Documentation Strategy** 📚: Use `go doc`, `go tool`, `go list` for Go packages.
- **English Only** 🇺🇸: Code and technical reasoning in English.

Expand Down Expand Up @@ -203,7 +205,7 @@ For detailed architecture, see [docs/CODEBASE_MAP.md](docs/CODEBASE_MAP.md).
## 🧹 Code Quality & Hygiene

### Linting & Static Analysis
- **Full Lint** 🔍: `go tool golangci-lint run --enable=unused --enable=unparam --enable=ineffassign --enable=goconst ./...`
- **Full Lint** 🔍: `go tool golangci-lint run --no-config --enable=unused --enable=unparam --enable=ineffassign --enable=goconst ./...`. The repository has no tracked linter configuration; `--no-config` prevents parent or home configuration from silently weakening local checks compared with CI.
- **Quick Check** ⚡: `go vet ./...` (Do not use `go build` for validation).
- **Compliance** ✅: **Never ignore lint warnings and fix them right away.**

Expand Down
31 changes: 21 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,22 @@
7. Optional greeting text can be shown immediately with the public CAPTCHA for direct joins, or after approval for join-request newcomers.

## Spam protection
1. Every previously untrusted author enters a per-chat message probation. Each new text, caption, or visible rich-message text is checked for at least three hours, and the first safe new message after the deadline is required for release:
- **Manual allowlist ("Indulgence") override**
- **Known spammers lookup** from local imports and online checks against LoLs bot and CAS/Combot
- **External quote heuristic** for obvious cross-chat spam patterns
- **LLM-powered binary classification** with a general or Jobs & HR profile plus chat-specific allowed and spam examples
2. If the message is considered spam, the user is either immediately banned or sent into community voting, depending on chat settings.
3. Chat users can report missed spam with `/voteban` or by mentioning the bot in reply to the message. Reports are rechecked by the LLM first, then either moderated immediately or sent to community voting without pre-deleting the original message.
4. Clean messages before the deadline remain bound for future edit checks. A distinct clean message after the deadline durably completes probation; commands and media without text start the clock but cannot complete it.
1. Message authors are identified per chat as either a user or a sender channel. Channel identity takes precedence over Telegram's technical sender. Group membership and reaction history do not grant message trust.
2. By default, three distinct safe new texts, captions, or visible rich-message texts grant 30 days of trust. After expiry, one safe new message renews it. Commands, bot mentions, edits, reactions, and media without meaningful text never advance or renew trust. Safe checks and their message bindings are committed atomically; duplicates cannot advance the count twice.
3. Manual user allowlists precede user banlist enforcement. Automatic trust never bypasses banlists or reports. Linked-channel posts and anonymous group administrators keep their existing exemptions. Without moderation rights, no provider-backed checks or punitive actions run.
4. The classifier receives the direct reply/quote, source post, and up to five recent replies from the same thread. Context is limited to 24 hours, 2,000 characters per saved message, and 8,000 additional characters per request. Unknown discussion threads use only the available reply chain; ordinary groups can use recent group messages. Conversation is untrusted evidence; only the current message is classified. External replies have no automatic spam verdict.
5. Before admission and after expiry, semantic edits are checked. During trust, edits to previously checked messages remain protected. Trusted messages are retained for context, edits update it, and bot deletions remove it with replay protection. Telegram does not reliably notify bots when users delete ordinary messages.
6. Automatic spam suspicions follow the chat's community-voting setting. Suspect messages are deleted; users may be muted while voting, but channels have no mute step. Channels are banned only after confirmation, immediately when voting is disabled. Open cases suspend admission; confirmed spam resets trust and its counter, while false positives restore any still-valid prior expiry. Voters must be actual chat members; channel owners are not inferred.
7. Report missed spam with `/voteban` or by mentioning the bot in reply. Reports are checked independently of automatic trust. Confirmed user reports and administrator decisions keep their immediate moderation path; automatic channel suspicions follow community voting.

On migration, previously effective member trust and completed probations receive 30 days from migration time. Active probations restart with zero safe messages; reaction-only non-member records grant no trust. Existing cases, votes, message bindings and pending actions are preserved. Downgrade is refused while channel cases/bindings or pending CAPTCHA context cleanup exist, to avoid losing author identity or unfinished work.

The accepted tradeoff is that an attacker can earn trust with three harmless messages. Subsequent new messages are then exempt from automatic LLM checks until expiry, while reports, protected edits and user banlists remain active.

## Admin panel
1. Run `/settings` in a group where the bot is an admin.
2. The bot sends a deep-link that opens a private admin panel for that chat.
3. From there you can configure gatekeeper, new-user message probation, community voting, the LLM moderation profile, allowed/spam examples, language, and manual not-spammer overrides.
3. From there you can configure gatekeeper, message author trust, community voting, the LLM moderation profile, allowed/spam examples, language, and manual not-spammer overrides.
4. The home screen includes a one-tap `Recommended Protection` preset and a compact 7-day protection summary.

## Installation
Expand Down Expand Up @@ -119,7 +122,9 @@ See [.env.example](.env.example) for a quick reference. `NGBOT_*` variables conf
| | `NG_SPAM_LOG_CHANNEL_USERNAME` | Channel for spam logging | | Any valid channel username |
| | `NG_SPAM_DEBUG_USER_ID` | User allowed to run diagnostics in private chat | `0` | Telegram user ID |
| | `NG_SPAM_VERBOSE` | Verbose in-chat notifications | `false` | `true`, `false` |
| | `NG_SPAM_MESSAGE_PROBATION_DURATION` | Minimum new-user message probation before a checked safe exit | `3h` | Any positive duration string |
| | `NG_SPAM_SAFE_MESSAGES_REQUIRED` | Distinct safe new messages for initial admission | `3` | Positive integer |
| | `NG_SPAM_AUTHOR_TRUST_DURATION` | Author trust lifetime and renewal period | `720h` | Positive duration |
| | `NG_SPAM_MESSAGE_PROBATION_DURATION` | Deprecated, accepted with a warning and ignored | Empty | Legacy duration string |
| | `NG_SPAM_VOTING_TIMEOUT` | Voting time limit | `5m` | Any valid duration string |
| | `NG_SPAM_MIN_VOTERS` | Minimum required voters | `2` | Any positive integer |
| | `NG_SPAM_MAX_VOTERS` | Maximum voters cap | `10` | Any positive integer |
Expand Down Expand Up @@ -214,3 +219,9 @@ This bot benefits from public anti-spam data shared with the community by:
Thank you to both projects for maintaining and sharing these community safety resources.

Feel free to add feature requests in issues.

### Author moderation diagnostics

Structured debug logs include `author_kind`, `author_id`, chat/message IDs, `trust_phase`, skip reason and classification outcome without message text. Existing provider usage logs retain token counts. Daily SQLite KV statistics use `stats:<chat_id>:<YYYY-MM-DD>:<metric>` keys: `author_check_initial`, `author_check_renewal`, `author_check_edit`, `author_check_pending_case`, `author_check_report`, `author_trust_granted`, and `author_trust_skipped`. Check counters include attempts; `llm_checked` counts successful message/report classifications, and existing `spam_confirmed` / `false_positive` counters record resolved cases. Compare equivalent traffic windows after rollout; local tests do not establish the cause or savings of a production incident.

Context cleanup runs through the existing periodic bounded retention job. Checked-message bindings contain only identity and timing metadata, and remain until the chat is removed. They survive trust expiry, renewal, resets and context deletion so that an old checked message cannot lose edit protection during a later trust period.
4 changes: 3 additions & 1 deletion compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ services:
NG_SPAM_LOG_CHANNEL_USERNAME: ${NG_SPAM_LOG_CHANNEL_USERNAME:-}
NG_SPAM_DEBUG_USER_ID: ${NG_SPAM_DEBUG_USER_ID:-0}
NG_SPAM_VERBOSE: ${NG_SPAM_VERBOSE:-false}
NG_SPAM_MESSAGE_PROBATION_DURATION: ${NG_SPAM_MESSAGE_PROBATION_DURATION:-3h}
NG_SPAM_SAFE_MESSAGES_REQUIRED: ${NG_SPAM_SAFE_MESSAGES_REQUIRED:-3}
NG_SPAM_AUTHOR_TRUST_DURATION: ${NG_SPAM_AUTHOR_TRUST_DURATION:-720h}
NG_SPAM_MESSAGE_PROBATION_DURATION: ${NG_SPAM_MESSAGE_PROBATION_DURATION:-}
NG_SPAM_VOTING_TIMEOUT: ${NG_SPAM_VOTING_TIMEOUT:-5m}
NG_SPAM_MIN_VOTERS: ${NG_SPAM_MIN_VOTERS:-2}
NG_SPAM_MAX_VOTERS: ${NG_SPAM_MAX_VOTERS:-10}
Expand Down
Loading