diff --git a/README.md b/README.md index 354dd41b..6332e843 100644 --- a/README.md +++ b/README.md @@ -1231,7 +1231,7 @@ Na exclusão, o serviço: Ficam as denúncias e as decisões dos moderadores (`moderation.json`, retenção em aberto na issue #38). Quem ainda é dono de uma página recebe `paginas-ativas` e precisa encerrá-la ou transferi-la antes. -O app promete que o diário sai dos semeadores em até 7 dias, então rode `mod apply --yes` agendado (a cada hora, como na seção anterior): ele assina as exclusões sem esperar o terminal. `GET /account/status?member=` responde `{ "linked": false, "deleted": "pendente" }` até a lista com a chave chegar ao semeador e o painel confirmá-la, e `"concluida"` depois; o app usa isso para avisar a pessoa. Cada exclusão ocupa uma das 4096 vagas da lista de bloqueio (`MAX_BLOCKED`). +O app promete que o diário sai dos semeadores em até 7 dias, então rode `mod apply --yes` agendado (a cada hora, como na seção anterior): ele assina as exclusões sem esperar o terminal. `GET /account/status?member=` responde `{ "linked": false, "deleted": "pendente" }` até a lista com a chave chegar ao semeador e o painel confirmá-la, e `"concluida"` depois; o app usa isso para avisar a pessoa. Cada exclusão ocupa para sempre uma das 16384 vagas da lista de bloqueio (`MAX_BLOCKED`), e hoje a lista está muito abaixo de 4096 chaves. Um nó de versão anterior aceita no máximo 4096: de 4097 a 5796 chaves, ele recusa a lista nova e continua aplicando a última lista válida que recebeu até ser atualizado; acima disso, faz o mesmo, mas o app e o PWA de versão anterior não concluem mais a sincronização (todo Sincronizar termina em `tempo esgotado: blocklist`) até serem atualizados. Veja `docs/protocol.md` seção 3, "Blocklist snapshot". A primeira exclusão grava `deleted` no `directory.json`: voltar a uma versão anterior a esta precisa do backup de antes dela, como na primeira página. @@ -1447,7 +1447,7 @@ Limites desta fase: - A identidade piloto da versão web mantém o certificado v1 (um diário não aceita um segundo certificado). Ela pode vincular a conta ao celular verificado e ganhar um @nome pelo registro de @nome, mas o certificado dela continua v1. - O @nome troca no máximo uma vez a cada 7 dias e um nome já usado por um membro não vai para outro; um nó antigo ou um seguidor que ainda não sincronizou pode mostrar por um tempo o nome anterior. - A lista de bloqueio é obedecida só por nós atualizados e honestos: um nó de antes da PR B2 não a aplica, um app modificado ainda pode falar direto com a conta bloqueada, e o app oficial da própria conta bloqueada guarda o diário e pode servi-lo a quem já conhece a chave de descoberta. A versão web sem rede aplica a lista que já guardou e só recebe uma nova pelo semeador. Bloquear não apaga nada do aparelho do autor. Veja `docs/verification.md` seção 13. -- A lista tem no máximo 4096 contas bloqueadas e 64 emissores revogados. Cada conexão troca no máximo uma lista a cada 10 s e é derrubada depois de 5 mensagens inválidas no canal `varanda/blocklist/1`. +- A lista tem no máximo 16384 contas bloqueadas (um nó de versão anterior aceita até 4096, veja "Excluir conta") e 64 emissores revogados. Cada conexão troca no máximo uma lista a cada 10 s e é derrubada depois de 5 mensagens inválidas no canal `varanda/blocklist/1`. - Só o nó hospedado denuncia (a versão web hospedada e o app instalado com a instalação): o app com configuração manual, o `varanda-serve` e a CLI não têm como assinar ou enviar uma denúncia (a folha avisa), e só quem vinculou a conta denuncia, até 20 vezes a cada 24 horas. O trecho citado da publicação é o que quem denuncia enviou e não é conferido pelo serviço, e a pessoa denunciada não vê nada. Veja `docs/verification.md` seção 14. - As sessões do painel acabam quando o serviço `account` reinicia. A lista que o painel mostra como publicada é o espelho verificado da última lista que o operador confirmou com `mod apply`, e o `blocklist.json` do operador é a que vale; "aplicado" é a palavra do operador, e a propagação segue as regras do `varanda/blocklist/1`. - Certificados de membro individuais não são revogáveis (bloqueie a chave do diário). A chave do emissor do serviço de conta é revogada com `mod revoke-issuer`; depois disso gere outra (`account key`), certifique (`assoc certify-issuer`), `account cert add`, reinicie o serviço e peça que os membros vinculem de novo, porque os certificados v2 antigos deixam de valer. A reemissão automática fica para uma PR futura. diff --git a/apps/cli/lib/account/mod-routes.js b/apps/cli/lib/account/mod-routes.js index bb73a5dd..60001a58 100644 --- a/apps/cli/lib/account/mod-routes.js +++ b/apps/cli/lib/account/mod-routes.js @@ -20,9 +20,9 @@ const { MAX_REPORTS, MAX_APPEALS, AI_ACTOR, OWNER_ACTOR, ANEXO_MAX_BYTES, ANEXO_ // Google account may report, so every report has an accountable author. // POST /account/report/anexo is the same report carrying one image or // one voice note (a private or group message, statements dm v2 / v3): -// the only route with -// a body above REPORT_BODY_MAX, and at most MAX_ANEXO_INFLIGHT such -// bodies are read at once. The image is stored as the reporter's word +// the only member route with a body above REPORT_BODY_MAX (the +// operator's api/ack is the other large one), and at most +// MAX_ANEXO_INFLIGHT such bodies are read at once. The image is stored as the reporter's word // and served to the panel from local disk only; it never reaches the AI // triage, a log or the seeder. // - a blocked member's appeal (POST /account/appeal/start, POST @@ -39,6 +39,11 @@ const { MAX_REPORTS, MAX_APPEALS, AI_ACTOR, OWNER_ACTOR, ANEXO_MAX_BYTES, ANEXO_ // cookie and no Origin rule; a signature by the association-certified // moderator key over a single-use server nonce, bound to this origin and // the exact body (packages/core/lib/mod-request.js), is the authority. +// api/ack carries a whole blocklist snapshot, so it is read like an +// anexo (a declared length, an idle cut, a deadline that grows with the +// length) but with no slot cap: a slot taken before the moderator +// signature is checked would let an anonymous sender keep the operator +// out, so ack reads share only the service's socket ceiling. // // Every server.js rule holds here: { error: token } bodies, no CORS, the // browser POST rules (exact Origin + JSON) on every cookie route, which with @@ -98,29 +103,43 @@ const REPORT_BODY_MAX = 8192 const ANEXO_B64_MAX = 4 * Math.ceil(ANEXO_MAX_BYTES / 3) const ANEXO_BODY_MAX = ANEXO_B64_MAX + REPORT_BODY_MAX const MAX_ANEXO_INFLIGHT = 4 -// Reading an anexo body is anonymous until its nonce and signature check, so -// a slot is never held by a sender that stops: a body with no chunk for -// ANEXO_IDLE_MS is cut, and every body has a deadline of ANEXO_READ_SLACK_MS -// plus its declared length at ANEXO_MIN_BITS_PER_MS (1 Mbit/s, a slow mobile -// link): a full ANEXO_BODY_MAX (about 1.4 MB, 11.3 s at that rate) would get -// about 13.3 s, a small image a couple of seconds. The service's -// requestTimeout (server.js) is the headers timeout plus ANEXO_READ_MAX_MS, -// so it never ends an anexo body before this deadline does. -const ANEXO_IDLE_MS = 5000 -const ANEXO_READ_SLACK_MS = 2000 -const ANEXO_MIN_BITS_PER_MS = 1000 -const ANEXO_READ_MAX_MS = ANEXO_READ_SLACK_MS + Math.ceil(ANEXO_BODY_MAX * 8 / ANEXO_MIN_BITS_PER_MS) +// Reading an anexo or an ack body is anonymous until its nonce and signature +// check, so a slot is never held by a sender that stops: a body with no +// chunk for LARGE_BODY_IDLE_MS is cut, and every body has a deadline of +// LARGE_BODY_SLACK_MS plus its declared length at LARGE_BODY_MIN_BITS_PER_MS +// (1 Mbit/s, a slow mobile link): a full ANEXO_BODY_MAX (about 1.4 MB, 11.3 s +// at that rate) gets about 13.3 s, a full ACK_BODY_MAX (1.5 MiB) about +// 14.6 s, a small image a couple of seconds. The service's requestTimeout +// (server.js) is the headers timeout plus LARGE_BODY_READ_MAX_MS, so it never +// ends such a body before this deadline does. +const LARGE_BODY_IDLE_MS = 5000 +const LARGE_BODY_SLACK_MS = 2000 +const LARGE_BODY_MIN_BITS_PER_MS = 1000 const DECIMAL_RE = /^[0-9]{1,10}$/ const B64_RE = /^[A-Za-z0-9+/]*={0,2}$/ // The image itself, served to a signed-in moderator: never a document. const ANEXO_CSP = "default-src 'none'; sandbox" -// A legal snapshot is at most 393216 bytes (blocklist.js MAX_SNAPSHOT_BYTES) -// plus at most 4096 change ids (mod-request.js MAX_CHANGE_IDS). -const ACK_BODY_MAX = 524288 +// A legal snapshot is at most 1179648 bytes (blocklist.js MAX_SNAPSHOT_BYTES) +// plus at most 4096 change ids of up to 67 JSON bytes each (mod-request.js +// MAX_CHANGE_IDS, 274432 bytes), plus the signed fields: 1.5 MiB leaves +// about 115 KB of room. +const ACK_BODY_MAX = 1572864 +// Ack reads have no slot cap of their own (see the header): only the +// service's socket ceiling (server.js MAX_SOCKETS) bounds them, as every +// route other than the anexo. +const ANEXO_READ_MAX_MS = largeBodyDeadlineMs(ANEXO_BODY_MAX) +const ACK_READ_MAX_MS = largeBodyDeadlineMs(ACK_BODY_MAX) +const LARGE_BODY_READ_MAX_MS = Math.max(ANEXO_READ_MAX_MS, ACK_READ_MAX_MS) const HEX64_RE = /^[0-9a-f]{64}$/ const TOKEN_RE = /^[A-Za-z0-9_-]{43}$/ // Report and change ids (moderation-store.js newId: 16 bytes base64url). const ID_RE = /^[A-Za-z0-9_-]{1,64}$/ + +// The read deadline of an anexo or ack body of `length` declared bytes. +function largeBodyDeadlineMs (length) { + return LARGE_BODY_SLACK_MS + Math.ceil(length * 8 / LARGE_BODY_MIN_BITS_PER_MS) +} + const REPORT_KINDS = new Set(['listing']) const ANUNCIO_KEYS = ['title', 'text', 'priceLabel', 'city', 'uf', 'kind', 'status'] // Google authorization codes: printable ASCII, bounded (as server.js). @@ -1239,23 +1258,23 @@ function createModRoutes ({ return readJsonBody(req, max) } - // The anexo body's declared length, or a refusal before any slot is - // taken: a chunked body (no Content-Length) could only be capped as it - // streams, and a length past the cap answers 413 unread. - function anexoLength (req, res) { + // A large body's declared length, or a refusal before any slot is taken: + // a chunked body (no Content-Length) could only be capped as it streams, + // and a length past `max` answers 413 unread. + function declaredLength (req, res, max) { const declared = req.headers['content-length'] if (req.headers['transfer-encoding'] !== undefined || typeof declared !== 'string' || !DECIMAL_RE.test(declared)) { res.setHeader('Connection', 'close') throw new RouteError('pedido-invalido', 411) } const length = Number(declared) - if (length > ANEXO_BODY_MAX) throw new RouteError('pedido-invalido', 413) + if (length > max) throw new RouteError('pedido-invalido', 413) return length } - // readJsonBody under the idle cut and the deadline (ANEXO_IDLE_MS): either - // destroys the request, which ends the read and so frees its slot. - function readAnexoBody (req, length) { + // readJsonBody under the idle cut and the deadline (largeBodyDeadlineMs): + // either destroys the request, which ends the read and so frees its slot. + function readLargeBody (req, length, max) { return new Promise((resolve, reject) => { let settled = false let idle = null @@ -1274,12 +1293,12 @@ function createModRoutes ({ } const onData = () => { clearTimeout(idle) - idle = setTimeout(cut, ANEXO_IDLE_MS) + idle = setTimeout(cut, LARGE_BODY_IDLE_MS) } - const deadline = setTimeout(cut, ANEXO_READ_SLACK_MS + Math.ceil(length * 8 / ANEXO_MIN_BITS_PER_MS)) - idle = setTimeout(cut, ANEXO_IDLE_MS) + const deadline = setTimeout(cut, largeBodyDeadlineMs(length)) + idle = setTimeout(cut, LARGE_BODY_IDLE_MS) req.on('data', onData) - readJsonBody(req, ANEXO_BODY_MAX, null).then((value) => finish(null, value), finish) + readJsonBody(req, max, null).then((value) => finish(null, value), finish) }) } @@ -1303,13 +1322,13 @@ function createModRoutes ({ if (post && p === '/account/report/anexo') { needModeration() checkPost(req) - const length = anexoLength(req, res) + const length = declaredLength(req, res, ANEXO_BODY_MAX) if (anexoInflight >= MAX_ANEXO_INFLIGHT) throw refuse('limite') anexoInflight++ // The slot is held until the whole report is handled, so the decode, // bounds check and hash of an image are bounded too, not only the read. try { - const body = await readAnexoBody(req, length) + const body = await readLargeBody(req, length, ANEXO_BODY_MAX) return await report(req, res, body, true) } finally { anexoInflight-- @@ -1333,7 +1352,11 @@ function createModRoutes ({ // the moderator signature over a server nonce is the authority). if (get && p === '/account/mod/api/challenge') return challenge(req, res, url) if (post && p === '/account/mod/api/pull') return pull(req, res, await postBody(req, undefined, { requireOrigin: false })) - if (post && p === '/account/mod/api/ack') return ack(req, res, await postBody(req, ACK_BODY_MAX, { requireOrigin: false })) + if (post && p === '/account/mod/api/ack') { + checkPost(req, { requireOrigin: false }) + const length = declaredLength(req, res, ACK_BODY_MAX) + return ack(req, res, await readLargeBody(req, length, ACK_BODY_MAX)) + } // Panel API. api/me answers 401 without a session even when the panel // is disabled: deploy/seeder/deploy.sh probes exactly that. @@ -1408,11 +1431,14 @@ module.exports = { REPORT_BODY_MAX, ANEXO_BODY_MAX, ANEXO_READ_MAX_MS, + ACK_READ_MAX_MS, + LARGE_BODY_READ_MAX_MS, MAX_ANEXO_INFLIGHT, ANEXO_LIST_LABEL, ANEXO_PROVENANCE, ANEXO_AUDIO_PROVENANCE, ACK_BODY_MAX, + largeBodyDeadlineMs, MAX_MOD_EMAILS, PANEL_CSP, parseModEmails, diff --git a/apps/cli/lib/account/server.js b/apps/cli/lib/account/server.js index f34d08d9..bf0b739f 100644 --- a/apps/cli/lib/account/server.js +++ b/apps/cli/lib/account/server.js @@ -9,7 +9,7 @@ const { isUsername, verifyAccountLink } = require('@varanda/core/lib/account-lin const { TERMS_VERSION, verifyTerms } = require('@varanda/core/lib/terms') const { verifyInvite } = require('@varanda/core/lib/invite') const { loadKeyPairFile, verifyIssuerCert, refuseSymlinkDir } = require('@varanda/core/lib/cert') -const { createModRoutes, ANEXO_READ_MAX_MS } = require('./mod-routes') +const { createModRoutes, LARGE_BODY_READ_MAX_MS } = require('./mod-routes') const { inviteId, CODE_RE } = require('./invites-store') const { createPushRoutes } = require('./push-routes') const { createSeenRoutes } = require('./seen-routes') @@ -76,12 +76,13 @@ const MAX_SOCKETS = 64 const DEFAULT_RATE = 20 const DEFAULT_BURST = 100 const HEADERS_TIMEOUT_MS = 4000 -// Long enough for the largest report anexo at the slowest rate its own -// deadline allows (mod-routes.js). Every other body is at most a few dozen -// KB and gets BODY_READ_MS from when its route starts reading it, the -// earlier request timeout, so only the anexo route holds a slow sender's -// socket past HEADERS_TIMEOUT_MS + BODY_READ_MS. -const REQUEST_TIMEOUT_MS = HEADERS_TIMEOUT_MS + ANEXO_READ_MAX_MS +// Long enough for the largest report anexo or operator ack at the slowest +// rate their own deadline allows (mod-routes.js LARGE_BODY_READ_MAX_MS, the +// larger of the two maxima). Every other body is at most a few dozen KB and +// gets BODY_READ_MS from when its route starts reading it, the earlier +// request timeout, so only those two routes hold a slow sender's socket +// past HEADERS_TIMEOUT_MS + BODY_READ_MS. +const REQUEST_TIMEOUT_MS = HEADERS_TIMEOUT_MS + LARGE_BODY_READ_MAX_MS const BODY_READ_MS = 10000 const CHECK_INTERVAL_MS = 1000 const MAX_BODY = 4096 @@ -1861,6 +1862,7 @@ module.exports = { MAX_SOCKETS, DEFAULT_RATE, DEFAULT_BURST, + HEADERS_TIMEOUT_MS, REQUEST_TIMEOUT_MS, BODY_READ_MS, KEY_FILE, diff --git a/apps/cli/lib/mod-apply.js b/apps/cli/lib/mod-apply.js index cad40ec2..1f0ad01f 100644 --- a/apps/cli/lib/mod-apply.js +++ b/apps/cli/lib/mod-apply.js @@ -55,10 +55,16 @@ const { jsonLine } = require('@varanda/core/lib/safe') const mod = require('./commands/mod') const REQUEST_TIMEOUT_MS = 10000 +// pull answers and ack bodies carry a whole snapshot (up to about 1.5 MiB): +// the service gives an ack body up to about 14.6 s to arrive +// (account/mod-routes.js ACK_READ_MAX_MS), so the whole request gets that +// plus room for the answer and the store write. +const TRANSFER_TIMEOUT_MS = 30000 const BODY_CAP = 524288 -// D1: at most MAX_PENDING (1000) changes of ~140 bytes plus one snapshot of -// up to MAX_SNAPSHOT_BYTES (393216) bytes. -const PULL_BODY_CAP = 1048576 +// D1: at most MAX_PENDING (1000) changes of at most ~220 bytes (64-character +// ids) plus one snapshot of up to MAX_SNAPSHOT_BYTES (1179648) bytes: 1.5 MiB +// leaves about 170 KB of room. +const PULL_BODY_CAP = 1572864 const MAX_CHANGES = 1000 // --yes (cron) signs at most this many new (human) blocks per run. const MAX_UNATTENDED_BLOCKS = 100 @@ -153,9 +159,9 @@ async function readCapped (res, cap) { // -> { status, value } with value the parsed JSON; throws UNAVAILABLE when // the server cannot be reached in time, INVALID for anything but bounded JSON. -async function request (fetch, url, { body = null, cap = BODY_CAP } = {}) { +async function request (fetch, url, { body = null, cap = BODY_CAP, timeoutMs = REQUEST_TIMEOUT_MS } = {}) { const controller = new AbortController() - const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS) + const timer = setTimeout(() => controller.abort(), timeoutMs) try { let res try { @@ -248,7 +254,7 @@ async function pull (ctx) { const nonce = await challenge(ctx, 'pull') const signature = sign(ctx, nonce, 'pull', canonicalBody('pull')) const body = JSON.stringify({ cert: ctx.certificate, nonce, signature }) - const { status, value } = await request(ctx.fetch, ctx.server + '/account/mod/api/pull', { body, cap: PULL_BODY_CAP }) + const { status, value } = await request(ctx.fetch, ctx.server + '/account/mod/api/pull', { body, cap: PULL_BODY_CAP, timeoutMs: TRANSFER_TIMEOUT_MS }) if (status !== 200) throw refused(value) const result = checkPull(value, ctx.association) if (result === null) throw new Error(INVALID) @@ -259,7 +265,7 @@ async function ack (ctx, changeIds, snapshot) { const nonce = await challenge(ctx, 'ack') const signature = sign(ctx, nonce, 'ack', canonicalBody('ack', { changeIds, snapshot })) const body = JSON.stringify({ cert: ctx.certificate, nonce, signature, changeIds, snapshot: canonicalSnapshot(snapshot) }) - const { status, value } = await request(ctx.fetch, ctx.server + '/account/mod/api/ack', { body }) + const { status, value } = await request(ctx.fetch, ctx.server + '/account/mod/api/ack', { body, timeoutMs: TRANSFER_TIMEOUT_MS }) if (status === 409 && isPlainObject(value) && value.error === 'defasado') { throw new Error('o servidor tem uma lista mais nova; restaure o backup') } @@ -436,4 +442,4 @@ async function signPublishAck (ctx, dir, changeIds, opts, done, edits = {}) { return done('aplicado', changeIds, snapshot, 'aplicado: seq ' + snapshot.seq + ', ' + snapshot.blocked.length + ' chaves bloqueadas') } -module.exports = { apply, checkServer, checkMode, REQUEST_TIMEOUT_MS, BODY_CAP, PULL_BODY_CAP, MAX_CHANGES, MAX_UNATTENDED_BLOCKS } +module.exports = { apply, checkServer, checkMode, REQUEST_TIMEOUT_MS, TRANSFER_TIMEOUT_MS, BODY_CAP, PULL_BODY_CAP, MAX_CHANGES, MAX_UNATTENDED_BLOCKS } diff --git a/apps/cli/test/account-mod.test.js b/apps/cli/test/account-mod.test.js index 4c59660b..5426ce28 100644 --- a/apps/cli/test/account-mod.test.js +++ b/apps/cli/test/account-mod.test.js @@ -17,15 +17,15 @@ const { spawnSync } = require('node:child_process') const b4a = require('b4a') const crypto = require('hypercore-crypto') const { createIssuerCert, createModeratorCert } = require('@varanda/core/lib/cert') -const { createSnapshot } = require('@varanda/core/lib/blocklist') +const { createSnapshot, MAX_BLOCKED, MAX_SNAPSHOT_BYTES } = require('@varanda/core/lib/blocklist') const { signReport, signListingReport } = require('@varanda/core/lib/report') const { signAppeal } = require('@varanda/core/lib/appeal') -const { signModRequest, canonicalBody } = require('@varanda/core/lib/mod-request') +const { signModRequest, canonicalBody, MAX_CHANGE_IDS } = require('@varanda/core/lib/mod-request') const { createGoogle } = require('../lib/account/google') const { openAccountStore } = require('../lib/account/store') const { openModerationStore } = require('../lib/account/moderation-store') -const { createAccountServer, COOKIE } = require('../lib/account/server') -const { SESSION_COOKIE, LOGIN_COOKIE, PANEL_CSP, parseModEmails } = require('../lib/account/mod-routes') +const { createAccountServer, COOKIE, BODY_READ_MS, HEADERS_TIMEOUT_MS, REQUEST_TIMEOUT_MS } = require('../lib/account/server') +const { SESSION_COOKIE, LOGIN_COOKIE, PANEL_CSP, parseModEmails, ACK_BODY_MAX, ACK_READ_MAX_MS, ANEXO_READ_MAX_MS, LARGE_BODY_READ_MAX_MS, largeBodyDeadlineMs } = require('../lib/account/mod-routes') const { createGroupCheck } = require('../lib/account/group-check') const { tmpDir } = require('./helpers/net') @@ -1213,6 +1213,112 @@ test('operator challenges: stateless, bound to moderator and op; anonymous flood noLeak(t, s) }) +test('operator ack and pull carry a list of MAX_BLOCKED keys (more than the old 4096)', async (t) => { + // The largest snapshot plus MAX_CHANGE_IDS ids of 64 characters (67 JSON + // bytes each) and the signed fields fit the ack body bound. + t.ok(ACK_BODY_MAX >= MAX_SNAPSHOT_BYTES + MAX_CHANGE_IDS * 67 + 1024, 'ACK_BODY_MAX covers the largest ack') + const s = await setup(t) + const modKeys = keys() + const modCert = createModeratorCert({ moderator: modKeys.hex, date: '2026-10-01T00:00:00.000Z' }, s.iss.assoc) + const challenge = async (op) => (await s.api('GET', '/account/mod/api/challenge?moderator=' + modKeys.hex + '&op=' + op)).body.nonce + const signed = (op, nonce, data) => signModRequest({ origin: ORIGIN, association: s.iss.association, moderator: modKeys.hex, nonce, op, body: canonicalBody(op, data) }, modKeys.kp) + const blocked = Array.from({ length: MAX_BLOCKED }, () => b4a.toString(crypto.randomBytes(32), 'hex')) + const snapshot = createSnapshot({ seq: 1, blocked }, modKeys.kp, modCert) + const changeIds = [] + const body = { cert: modCert, nonce: await challenge('ack'), changeIds, snapshot } + body.signature = signed('ack', body.nonce, { changeIds, snapshot }) + t.ok(Buffer.byteLength(JSON.stringify(body)) > 524288, 'over the old ack bound') + const acked = await s.api('POST', '/account/mod/api/ack', { origin: null, body }) + t.is(acked.status, 200, acked.text && acked.text.slice(0, 200)) + t.alike(acked.body, { ok: true, seq: 1 }) + t.is(s.moderation.stats().blocked, MAX_BLOCKED) + const nonce = await challenge('pull') + const pulled = await s.api('POST', '/account/mod/api/pull', { origin: null, body: { cert: modCert, nonce, signature: signed('pull', nonce) } }) + t.is(pulled.status, 200) + t.is(pulled.body.published.snapshot.blocked.length, MAX_BLOCKED) +}) + +// An operator POST over a raw socket that sends `first` and then nothing +// more (or ends there with `end`). -> { req, done } with done resolving to +// { status, body, connection }, or to 'cut' when the server drops the +// connection without an answer. +function rawModPost (s, pathname, headers, first, { end = false } = {}) { + const req = http.request(s.base + pathname, { method: 'POST', headers: { 'content-type': 'application/json', ...headers } }) + const done = new Promise((resolve) => { + req.on('response', (res) => { + const chunks = [] + res.on('data', (c) => chunks.push(c)) + res.on('end', () => { + let body = null + try { body = JSON.parse(Buffer.concat(chunks).toString('utf8')) } catch {} + resolve({ status: res.statusCode, body, connection: res.headers.connection }) + }) + res.on('error', () => resolve({ status: res.statusCode, body: null, connection: res.headers.connection })) + }) + req.on('error', () => resolve('cut')) + }) + req.write(first) + if (end) req.end() + return { req, done } +} + +test('operator ack: read deadline grows with the declared length, past the 10 s of other bodies', (t) => { + t.is(ACK_READ_MAX_MS, largeBodyDeadlineMs(ACK_BODY_MAX)) + t.ok(ACK_READ_MAX_MS > BODY_READ_MS, 'a full ack body gets more than BODY_READ_MS: ' + ACK_READ_MAX_MS + ' ms') + // 1 Mbit/s: a full ack body takes about 12.6 s on the wire. + t.ok(ACK_READ_MAX_MS >= Math.ceil(ACK_BODY_MAX * 8 / 1000), 'covers the body at 1 Mbit/s') + t.ok(largeBodyDeadlineMs(ACK_BODY_MAX) > largeBodyDeadlineMs(ACK_BODY_MAX / 2), 'grows with the length') + t.ok(largeBodyDeadlineMs(8000) < BODY_READ_MS, 'a small body gets a short deadline') + t.is(LARGE_BODY_READ_MAX_MS, Math.max(ANEXO_READ_MAX_MS, ACK_READ_MAX_MS)) + t.ok(REQUEST_TIMEOUT_MS >= HEADERS_TIMEOUT_MS + ANEXO_READ_MAX_MS, 'requestTimeout never cuts an anexo first') + t.ok(REQUEST_TIMEOUT_MS >= HEADERS_TIMEOUT_MS + ACK_READ_MAX_MS, 'requestTimeout never cuts an ack first') +}) + +test('operator ack: a body without Content-Length is refused, one past ACK_BODY_MAX answers 413 unread', async (t) => { + const s = await setup(t) + const chunked = await rawModPost(s, '/account/mod/api/ack', {}, '{"changeIds":[]}', { end: true }).done + t.alike(chunked, { status: 411, body: { error: 'pedido-invalido' }, connection: 'close' }, 'chunked body refused') + // Only the headers and one byte: the answer comes without the rest. + const big = rawModPost(s, '/account/mod/api/ack', { 'content-length': ACK_BODY_MAX + 1 }, '{') + const over = await big.done + big.req.destroy() + t.is(over.status, 413) + t.alike(over.body, { error: 'pedido-invalido' }) + t.is(s.moderation.stats().blocked, 0, 'nothing acked') +}) + +test('operator ack: stalled anonymous ack bodies never keep a signed ack out', async (t) => { + const s = await setup(t) + const modKeys = keys() + const modCert = createModeratorCert({ moderator: modKeys.hex, date: '2026-10-01T00:00:00.000Z' }, s.iss.assoc) + // Three bodies that declared a large length and stopped after a few bytes. + const stalled = [] + for (let i = 0; i < 3; i++) stalled.push(rawModPost(s, '/account/mod/api/ack', { 'content-length': 1000000 }, '{"changeIds":[')) + t.teardown(() => { for (const x of stalled) x.req.destroy() }) + const nonce = (await s.api('GET', '/account/mod/api/challenge?moderator=' + modKeys.hex + '&op=ack')).body.nonce + const snapshot = createSnapshot({ seq: 1, blocked: [s.ana.hex] }, modKeys.kp, modCert) + const changeIds = [] + const signature = signModRequest({ origin: ORIGIN, association: s.iss.association, moderator: modKeys.hex, nonce, op: 'ack', body: canonicalBody('ack', { changeIds, snapshot }) }, modKeys.kp) + const acked = await s.api('POST', '/account/mod/api/ack', { origin: null, body: { cert: modCert, nonce, changeIds, snapshot, signature } }) + t.is(acked.status, 200, acked.text && acked.text.slice(0, 200)) + t.alike(acked.body, { ok: true, seq: 1 }) + for (const x of stalled) t.ok(!x.req.destroyed, 'the stalled bodies were still open') +}) + +test('operator ack: a body that drips past its own deadline is cut long before 10 s', async (t) => { + const s = await setup(t) + const started = Date.now() + // 8000 bytes declared: a deadline of 2 s plus 64 ms. A byte every 400 ms + // keeps the idle cut away, so only the deadline can end it. + const drip = rawModPost(s, '/account/mod/api/ack', { 'content-length': 8000 }, '{') + const timer = setInterval(() => { if (!drip.req.destroyed) drip.req.write(' ') }, 400) + const end = await drip.done + clearInterval(timer) + const elapsed = Date.now() - started + t.ok(end === 'cut' || end.status === 408, 'cut by the server') + t.ok(elapsed >= 1900 && elapsed < 4500, 'at its deadline (' + elapsed + ' ms)') +}) + // The login cookie with one base64url character changed in the middle (the // sealed bytes change, so the auth tag no longer verifies). function tamper (value) { diff --git a/apps/cli/test/mod-apply.test.js b/apps/cli/test/mod-apply.test.js index 999fd323..3c5f91b3 100644 --- a/apps/cli/test/mod-apply.test.js +++ b/apps/cli/test/mod-apply.test.js @@ -17,7 +17,8 @@ const { lockDataDir } = require('@varanda/core/lib/store') const { loadKeyPairFile } = require('@varanda/core/lib/cert') const { verifySnapshot, createSnapshot, MAX_BLOCKED } = require('@varanda/core/lib/blocklist') const { verifyModRequest, canonicalBody } = require('@varanda/core/lib/mod-request') -const { apply, checkServer, MAX_UNATTENDED_BLOCKS } = require('../lib/mod-apply') +const { apply, checkServer, MAX_UNATTENDED_BLOCKS, PULL_BODY_CAP, REQUEST_TIMEOUT_MS, TRANSFER_TIMEOUT_MS } = require('../lib/mod-apply') +const { ACK_READ_MAX_MS } = require('../lib/account/mod-routes') const { testnet, tmpDir } = require('./helpers/net') const BIN = path.join(__dirname, '..', 'bin', 'varanda.js') @@ -446,6 +447,7 @@ test('a full list takes an unblock and a replacement block in one run', async (t ], published: { snapshot: full } } + t.ok(Buffer.byteLength(JSON.stringify(srv.pull)) > 1048576, 'the pull answer is over the old cap') const pub = publisher(true) const res = await run(ctx, srv, pub.publish) t.is(res.error, null, res.error && res.error.message) @@ -477,6 +479,42 @@ test('a published snapshot ahead of the local list exits writing nothing', async t.is(srv.byPath('/account/mod/api/ack').length, 0) }) +test('pull and ack wait longer than the service gives an ack body; the challenges keep 10 s', async (t) => { + t.is(REQUEST_TIMEOUT_MS, 10000) + t.ok(TRANSFER_TIMEOUT_MS >= ACK_READ_MAX_MS + 5000, 'past the ack read deadline with room for the answer') + await testnet(t) + const ctx = setup(t) + const first = seedList(t, ctx.modDir, X) + const srv = await fakeServer(t) + srv.pull = { changes: [{ id: 'c1', op: 'block', member: A, at: '2026-10-24T10:00:01.000Z', actor: 'pessoa' }], published: { snapshot: first } } + // request() arms its abort timer right before it calls fetch: the last + // setTimeout delay seen when fetch is called is that request's timeout. + const timeouts = [] + let lastDelay = null + const realSetTimeout = globalThis.setTimeout + globalThis.setTimeout = function (fn, ms, ...rest) { + lastDelay = ms + return realSetTimeout.call(this, fn, ms, ...rest) + } + const fetch = (url, init) => { + timeouts.push([new URL(url).pathname, lastDelay]) + return globalThis.fetch(url, init) + } + let res + try { + res = await run(ctx, srv, publisher(true).publish, { fetch }) + } finally { + globalThis.setTimeout = realSetTimeout + } + t.is(res.error, null, res.error && res.error.message) + t.alike(timeouts, [ + ['/account/mod/api/challenge', REQUEST_TIMEOUT_MS], + ['/account/mod/api/pull', TRANSFER_TIMEOUT_MS], + ['/account/mod/api/challenge', REQUEST_TIMEOUT_MS], + ['/account/mod/api/ack', TRANSFER_TIMEOUT_MS] + ]) +}) + test('malformed answers and a stale ack exit 1', async (t) => { await testnet(t) const ctx = setup(t) @@ -496,7 +534,7 @@ test('malformed answers and a stale ack exit 1', async (t) => { { status: 200, body: { changes: new Array(1001).fill(0).map((_, i) => ({ ...change, id: 'c' + i })), published: { snapshot: null } } }, { status: 200, body: '{"changes":[],', type: 'application/json' }, { status: 200, body: '{"changes":[],"published":{"snapshot":null}}', type: 'text/html' }, - { status: 200, body: JSON.stringify({ changes: [], published: { snapshot: null }, pad: 'x'.repeat(1048576) }) }, + { status: 200, body: JSON.stringify({ changes: [], published: { snapshot: null }, pad: 'x'.repeat(PULL_BODY_CAP) }) }, { status: 403, body: { error: 'nao-autorizado', detail: 1 } } ] for (const answer of bad) { diff --git a/apps/cli/test/moderation-store.test.js b/apps/cli/test/moderation-store.test.js index 766df001..a2edaf85 100644 --- a/apps/cli/test/moderation-store.test.js +++ b/apps/cli/test/moderation-store.test.js @@ -814,12 +814,18 @@ test('strict open: every report field, change and event is checked, never filter async function fullList (t) { const s = setup(t) const keys = [] - for (let i = 0; i < MAX_BLOCKED; i++) keys.push(key()) + for (let i = 0; i < MAX_BLOCKED; i++) keys.push(b4a.toString(crypto.randomBytes(32), 'hex')) await s.store.ack({ changeIds: [], snapshot: s.m.snap(1, keys), moderatorCert: s.m.cert }) t.is(s.store.stats().blocked, MAX_BLOCKED) return { ...s, keys } } +test('capacity: the mirror holds MAX_BLOCKED (16384) keys and reopens from disk', async (t) => { + t.is(MAX_BLOCKED, 16384) + const s = await fullList(t) + t.is(s.open().stats().blocked, MAX_BLOCKED, 'moderation.json read back whole') +}) + async function refusedUnchanged (t, s, p, msg) { const before = fs.readFileSync(path.join(s.dir, FILE), 'utf8') const pendingBefore = s.store.pending().changes diff --git a/docs/changelog/feat-blocklist-cap-16384.md b/docs/changelog/feat-blocklist-cap-16384.md new file mode 100644 index 00000000..6e8320ec --- /dev/null +++ b/docs/changelog/feat-blocklist-cap-16384.md @@ -0,0 +1,3 @@ +### Alterado + +- A lista de bloqueio assinada passa a aceitar até 16384 chaves (antes 4096), e o tamanho máximo dela passa de 393216 para 1179648 bytes. Cada exclusão de conta deixa a chave na lista para sempre, então a lista só cresce. Compatibilidade, decidida pela manutenção: um nó de versão anterior lê uma lista de 4097 a 5796 chaves como inválida e continua aplicando a última lista válida que recebeu; acima disso (o antigo teto de 393216 bytes), o app e o PWA de versão anterior não concluem mais a sincronização (`tempo esgotado: blocklist` em todo Sincronizar) até serem atualizados. Hoje a lista está muito abaixo de 4096 chaves. O corpo do `ack` da moderação e a resposta do `pull` do `mod apply` sobem para 1,5 MiB, e os prazos de envio e de espera crescem com eles. Parte da #31. Verificação em `docs/verification/feat-blocklist-cap-16384.md`. diff --git a/docs/protocol.md b/docs/protocol.md index c1ffec4d..53ecd6ea 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -662,11 +662,18 @@ The moderator signs numbered snapshots of the blocked diary keys and the revoked } ``` -Exactly these 9 keys, in this order: `v` (`1`), `seq` (safe integer >= 1), `date` (canonical ISO), `blocked` (array of at most `MAX_BLOCKED = 4096` (= `MAX_INVENTORY`, section 5) lowercase hex64 keys, strictly increasing, i.e. sorted lexicographically and distinct), `revokedIssuers` (the same rule with at most `MAX_REVOKED_ISSUERS = 64` issuer keys), `moderator` (lowercase hex64), `moderatorDate` (canonical ISO), `moderatorSignature` and `signature` (128 lowercase hex). Anything else (including uppercase, unsorted, duplicate or too many keys) is `'malformed'`. `moderator`, `moderatorDate` and `moderatorSignature` are the moderator certificate's `moderator`, `date` and `signature`; the association is not repeated: the verifier supplies it. +Exactly these 9 keys, in this order: `v` (`1`), `seq` (safe integer >= 1), `date` (canonical ISO), `blocked` (array of at most `MAX_BLOCKED = 16384` (>= `MAX_INVENTORY`, section 5, so a seeder never serves more logs than one snapshot can block) lowercase hex64 keys, strictly increasing, i.e. sorted lexicographically and distinct), `revokedIssuers` (the same rule with at most `MAX_REVOKED_ISSUERS = 64` issuer keys), `moderator` (lowercase hex64), `moderatorDate` (canonical ISO), `moderatorSignature` and `signature` (128 lowercase hex). Anything else (including uppercase, unsorted, duplicate or too many keys) is `'malformed'`. `moderator`, `moderatorDate` and `moderatorSignature` are the moderator certificate's `moderator`, `date` and `signature`; the association is not repeated: the verifier supplies it. `payload` = UTF-8 bytes of `'varanda-blocklist-v1'`, `seq` in decimal, `date`, the `blocked` keys joined by `','` (`''` when empty), the `revokedIssuers` keys joined by `','` (`''` when empty), `moderator`, `moderatorDate`, `moderatorSignature` joined by a single LF (no trailing LF). `signature` = `sign(payload, moderatorSecretKey)`. -Size: `MAX_SNAPSHOT_BYTES = 393216`; the JSON of the largest valid snapshot (4096 blocked keys, 64 revoked issuers, expanded-year dates, `seq = Number.MAX_SAFE_INTEGER`) is under it (`packages/core/test/blocklist.test.js`), and far under the seeder's `MAX_EXTERNAL_FRAME` (section 4). +Size: `MAX_SNAPSHOT_BYTES = 1179648` (1.125 MiB); the JSON of the largest valid snapshot (16384 blocked keys, 64 revoked issuers, expanded-year dates, `seq = Number.MAX_SAFE_INTEGER`, 1102534 bytes) is under it (`packages/core/test/blocklist.test.js`), and far under the seeder's `MAX_EXTERNAL_FRAME` (section 4). + +Compatibility: `MAX_BLOCKED` was 4096, and `MAX_SNAPSHOT_BYTES` and `MAX_BLOCKLIST_MESSAGE` were 393216, before. Three ranges for a node on an older build: +- Up to 4096 keys: every build reads the list the same. +- From 4097 to 5796 keys: the snapshot message still fits the old 393216-byte bound (5796 is the worst case: 64 revoked issuers, expanded-year dates, `seq = Number.MAX_SAFE_INTEGER`; with no revoked issuer, `seq = 1` and current dates, 5861 keys still fit). The older node verifies the list as `'malformed'`, refuses it and keeps enforcing its last valid list until it updates; its sync settles as usual. +- Above that: the older node counts the message as a bad message at the raw size check of the channel below and never offers it, so it keeps its last valid list too. An older browser node (`packages/web-member`, which is also the native app's worklet, and any cached PWA) then never meets the blocklist goal of its sync (`blocklistSettled` in `packages/web-member/lib/sync.js`): every once run ends with `tempo esgotado: blocklist` and continuous sync never reports ready, until its build updates. An older core node has no such goal. + +A deleted account keeps its key in the list for good ("Account deletion" Limits, below), so the list only grows; it is far below 4096 keys today. - `snapshotPayload(snapshot)` -> Buffer; `validateSnapshotShape(snapshot)` -> `null | 'malformed'`. - `createSnapshot({ seq, date = new Date().toISOString(), blocked = [], revokedIssuers = [] }, moderatorKeyPair, moderatorCert)` -> snapshot. Keys are accepted in any case, order and with repeats; they are stored lowercase, sorted and distinct. TypeError for a bad `seq`/`date`/key, more than the limit, a bad key pair, or a moderator certificate that is malformed, does not verify for its own association or names another key. @@ -679,7 +686,7 @@ Size: `MAX_SNAPSHOT_BYTES = 393216`; the JSON of the largest valid snapshot (409 ### Blocklist channel (`'varanda/blocklist/1'`, `packages/core/lib/blocklist-net.js`) -Carries the snapshot above between nodes on the SAME protomux that replication and the admission channel use (`socket.noiseStream.userData`). Messages are UTF-8 JSON (compact-encoding `c.buffer`), at most `MAX_BLOCKLIST_MESSAGE = 393216` bytes (the same bound as `MAX_SNAPSHOT_BYTES`): +Carries the snapshot above between nodes on the SAME protomux that replication and the admission channel use (`socket.noiseStream.userData`). Messages are UTF-8 JSON (compact-encoding `c.buffer`), at most `MAX_BLOCKLIST_MESSAGE = 1179648` bytes (the same bound as `MAX_SNAPSHOT_BYTES`): - `have`: exactly `{ "type": "have", "moderatorDate": , "seq": = 0>, "sig": }`: the ordering key of the snapshot the sender holds (`compareSnapshots`, above) plus a prefix that identifies its contents. `moderatorDate` `''` requires `sig` `''`; `sig` takes no part in the order. It lets the operator's publisher tell "the seeder holds exactly this list" from "the seeder holds another list with the same (`moderatorDate`, `seq`)" (after restoring an old backup of the moderation directory): the first is a delivery, the second a conflict (`{ delivered: false, conflict: true }`, the command exits 1). A `have` over `MAX_HAVE_BYTES = 256` bytes is a bad message. - `snapshot`: exactly `{ "type": "snapshot", "snapshot": BlocklistSnapshot }`. @@ -857,7 +864,7 @@ A member deletes the account from the app (`apps/cli/lib/account/delete-routes.j - What the service deletes: the link (sub, e-mail, certificate, records, invite id), `terms.json`, `age.json`, `push.json`, `recovery.json` (each entry leaves the store's usual tombstone with the public key, 30 days), the invites the member issued (`invites.json`), the surprise-invite marks (`invite-drops.json`), the encrypted seen blob (`seen/`, through the store's compare-and-swap; a conflict waits for the next sweep) and the stored passkey keys with any waiting deletion (`passkeys.json`, a 30-day tombstone with the public key). What stays: the deleted row (public key and date) and every @nome the key held, burned (`usedNames`); reports and decisions in `moderation.json` (retention open, issue #38); the access logs, if they apply. The deletion sweep runs after every request, at every start and every hour, so a store that could not be purged at once (unavailable, a failed write, or rows a stale writer's union merge brought back) is purged later; logs carry counts only. - A page delegate (section 12) whose diary was deleted is refused (`nao-vinculado`) before the block reaches it. The age routes refuse a deleted key (`POST /account/age/session` and `/result` answer 409 `conta-excluida`; a provider session opened before the deletion is closed there), so it never pays for, nor shows anything to, the provider and never gets a band again. Pages where the member was only an admin keep the key in their roster until the owner removes it (that needs the owner's signed roster); the service refuses every action of the deleted key at once. - On the device, after a deletion that ran now, the app wipes the account as `identity.reset()` does and keeps only `{ v: 1, member, at }` (public key and date) to show, at Entrar, when the deletion completed. A scheduled deletion wipes nothing: every signed-in device shows the date with a way to cancel, and once it runs each device meets the notice below. Another device still signed in with the same passkey meets a notice when it reads `/account/status`, and erases the account there. A diary key cannot be revoked (the passkey derives it again), so the account service refuses that key for good: `start` answers `conta-excluida` and the directory never links it, reserves it or makes it a page. -- Limits: a key is trusted on first use, at sign-up: someone who controls the person's device during the hour after the link could enrol a forged authenticator instead. An unlocked session holds the diary key, so it can always schedule a deletion; the person then sees it on every signed-in device (and in a push, when notices are on) and can cancel it for 7 days, but a person who opens no device in that time loses the account. Extra passkeys are not enrolled, so a person whose only stored passkey is gone waits the 7 days. iOS may give no attestation object at creation (`rawAttestationObject` is optional), and the iOS origin is expected to be `https://`: both are device checks (`docs/verification.md` section 116). The list holds at most 4096 keys (`MAX_BLOCKED`), and every deletion keeps one; copies on modified apps and devices that ignore the list are not reached; `directoryFloor` (`apps/cli/lib/account/age-check-state.js`) is read from the live links only, so after a restart it no longer counts a deleted member who had no age band, which matters only until the block reaches that diary. +- Limits: a key is trusted on first use, at sign-up: someone who controls the person's device during the hour after the link could enrol a forged authenticator instead. An unlocked session holds the diary key, so it can always schedule a deletion; the person then sees it on every signed-in device (and in a push, when notices are on) and can cancel it for 7 days, but a person who opens no device in that time loses the account. Extra passkeys are not enrolled, so a person whose only stored passkey is gone waits the 7 days. iOS may give no attestation object at creation (`rawAttestationObject` is optional), and the iOS origin is expected to be `https://`: both are device checks (`docs/verification.md` section 116). The list holds at most 16384 keys (`MAX_BLOCKED`), and every deletion keeps one for good: a node on an older build refuses a list of more than 4096 keys, and an older browser node stops settling its sync above 5796 ("Blocklist snapshot" Compatibility, above); the list is far below 4096 keys today; copies on modified apps and devices that ignore the list are not reached; `directoryFloor` (`apps/cli/lib/account/age-check-state.js`) is read from the live links only, so after a restart it no longer counts a deleted member who had no age band, which matters only until the block reaches that diary. ### Enforcement: the `'blocked'` and `'revoked-issuer'` reasons at every gate @@ -1754,7 +1761,7 @@ The same listener answers the engagement index of section 5 ("Seeder engagement - `packages/core/lib/phone.js` (section 3 "Reports and operator requests", the phone-verification request; pure: `b4a` and `hypercore-crypto` only): `PREFIX` = `'varanda-phone-v1'`, `ACTIONS` = `['send', 'check', 'status']` (frozen), `NONCE_RE` = `/^[A-Za-z0-9_-]{43,128}$/`, `BR_MOBILE_RE` = `/^\+55[1-9][1-9]9[0-9]{8}$/` (a Brazilian mobile in E.164; the number is never a signed field), `isBrMobile(value)`, `phonePayload({ action, nonce, member, association })` -> the UTF-8 bytes (TypeError on a bad shape), `signPhone(fields, keyPair)` -> lowercase hex128, `verifyPhone(fields, signatureHex)` -> `{ ok: true } | { ok: false, reason: 'malformed' | 'bad-signature' }`. `verifyPhone` never throws. - `packages/core/lib/phone-proof.js` (section 3 "Phone attestations"; pure: `b4a`, `hypercore-crypto` and `./cert`): `PHONE_VERIFIED_PREFIX`, `PHONE_SINCE_PREFIX`, `MAX_PHONE_PROOF_BYTES`, `phoneVerifiedPayload(att)` / `phoneSincePayload(att)` -> Buffer, `createPhoneVerified({ member, date }, issuerKeyPair, issuerCert)`, `createPhoneSince({ since, date }, issuerKeyPair, issuerCert)`, `validatePhoneVerifiedShape(att)` / `validatePhoneSinceShape(att)` -> `null | 'malformed'`, `verifyPhoneVerified(att, memberHex, associationHex, { revokedIssuers })` and `verifyPhoneSince(att, associationHex, { revokedIssuers })` -> `{ ok, reason }`. The validators and verifiers never throw. - `packages/core/lib/blocklist-net.js` (B2): `PROTOCOL`, `MAX_BLOCKLIST_MESSAGE`, `MIN_SNAPSHOT_INTERVAL_MS`, `MAX_BAD_BLOCKLIST`, `MAX_RETRY_DELAY_MS`, `MESSAGE_BURST`, `MESSAGE_RATE`, `MAX_HAVE_BYTES`, `validateBlocklistMessage`, `compareHave`, `haveOf`, `attachBlocklist(socket, { current, offer, compare, onhave, log, maxBytes, minIntervalMs, maxBad, messageBurst, messageRate })` -> `{ peerKey, socket, remoteHave, badMessages, supported, announce(), send, close() }` (section 3 "Blocklist channel"). `packages/core/lib/store.js` also exports `readBlocklist`, `writeBlocklist`, `replicaStored` and `lockDataDir`. -- `packages/core/lib/blocklist.js` (B2, pure): `PREFIX`, `SNAPSHOT_KEYS`, `MAX_BLOCKED = 4096`, `MAX_REVOKED_ISSUERS = 64`, `MAX_SNAPSHOT_BYTES = 393216`, `snapshotPayload`, `validateSnapshotShape`, `createSnapshot`, `verifySnapshot`, `compareSnapshots`, `acceptSnapshot`, `blockedSet`, `revokedIssuerSet`, `isBlocked`, `createBlocklistState` (section 3 "Blocklist snapshot"). +- `packages/core/lib/blocklist.js` (B2, pure): `PREFIX`, `SNAPSHOT_KEYS`, `MAX_BLOCKED = 16384`, `MAX_REVOKED_ISSUERS = 64`, `MAX_SNAPSHOT_BYTES = 1179648`, `snapshotPayload`, `validateSnapshotShape`, `createSnapshot`, `verifySnapshot`, `compareSnapshots`, `acceptSnapshot`, `blockedSet`, `revokedIssuerSet`, `isBlocked`, `createBlocklistState` (section 3 "Blocklist snapshot"). - `packages/core/lib/username.js`: `latestUsernameRecord(entries, memberHex, associationHex, { revokedIssuers } = {})` -> plain-data copy of the highest-seq shape-valid `username` record of that member if `verifyUsernameRecord` accepts it for that member and association (only that one candidate is verified, no fallback), or `null`; `latestUsername(entries, memberHex, associationHex, { revokedIssuers } = {})` -> its `username` or `null` (a record from a revoked issuer is invalid). Never throw. - `packages/core/lib/assoc.js`: `initAssociation(dir)` -> `{ publicKey }`, `showAssociation(dir)` -> `{ publicKey }`, `issueCertificate(dir, { member, ageBand, method })` -> Certificate, `issueSeederCertificate(dir, { seeder })` -> SeederCertificate (appends ONE `{ "seeder": hex64, "date": ISO }` line + LF to `/seeders.jsonl`). - `packages/core/lib/records.js`: `MAX_NAME`, `MAX_BIO`, `MAX_TEXT`, `MAX_BLOCK_BYTES = 16384`, `MAX_LOG_BLOCKS = 100000`, `MAX_LOG_BYTES = 67108864` (S3), `validRecordCopy(record, seq)` -> plain-data copy of a valid record (`structuredClone` when the runtime has it, else a plain-data walker that copies only null, booleans, numbers, strings, arrays and plain objects and refuses everything else: functions, `undefined`, BigInt, cycles, accessors that throw; validated on the copy, never throws) or `null`, `nowIso()`, `isHex64(s)`, `isCanonicalIso(s)` -> boolean (R4.3, never throws), `decodeBlock(buf)` -> plain object | `null` (R4.1), `memberCertRecord(cert)`, `profileRecord({ name, bio, avatar, at })` (`avatar` added only when given and not `null`), `profileRecords({ name, bio, avatar, at })` -> `[plain]` or the compatibility pair `[plain, withAvatar]` (section 2), `validAvatar(d)` -> boolean (never throws), the avatar constants `AVATAR_SIDE`, `MAX_AVATAR_BLOCKS`, `MAX_AVATAR_BYTES`, `AVATAR_MIMES` and `PROFILE_AVATAR_KEYS`, `postRecord({ text, replyTo, at, media })` (`media` added only when given), `mediaCoreRecord({ key, at })`, `followRecord({ key, on, at })`, `validDescriptor(d)` -> boolean (never throws), the media constants `MEDIA_BLOCK_BYTES`, `MAX_MEDIA_BLOCKS`, `MAX_MEDIA_BYTES`, `MAX_MEDIA_CORES`, `MAX_MEDIA_PER_POST`, `MAX_DELETE_RANGES`, `MAX_MEDIA_ITEM_BLOCKS`, `MEDIA_MIMES`, `MAX_MEDIA_DIMENSION` (section 2; a listing checks its media against `feira.MAX_LISTING_MEDIA` of `@varanda/rpc` instead of `MAX_MEDIA_PER_POST`), `deleteRecords({ seq, at, media })` -> array of `delete` records, one per chunk of `MAX_DELETE_RANGES` ranges (section 2, "Deletion"), `validateRecord(record, seq)` -> `null | reason` (`validateRecord(null, seq)` returns `'malformed'`; never throws, R4.3), `parseRef(str)` -> `{ author, seq } | null`, `formatRef({ author, seq })`, `latestProfile(entries)` -> `{ name, bio, avatar } | null` (`avatar` a fresh descriptor copy or `null`), `genesisCert(entries)` -> `cert | null` (v1 or v2); `entries` = `[{ seq, record }]`. records.js does not import cert.js: it duplicates the v1/v2 certificate and username-record shape rules (a parity test in `packages/core/test/records.test.js` feeds both validators the same objects). diff --git a/docs/verification/feat-blocklist-cap-16384.md b/docs/verification/feat-blocklist-cap-16384.md new file mode 100644 index 00000000..451a140a --- /dev/null +++ b/docs/verification/feat-blocklist-cap-16384.md @@ -0,0 +1,53 @@ +# feat/blocklist-cap-16384 + +Parte de #31. Cada exclusão de conta deixa a chave do diário na lista de bloqueio para sempre (certificados de membro não expiram), então o teto de 4096 chaves seria atingido com o tempo. Decisão da manutenção (2026-11-17): subir para 16384. + +## O que mudou + +- `packages/core/lib/blocklist.js`: `MAX_BLOCKED` 4096 -> 16384 e `MAX_SNAPSHOT_BYTES` 393216 -> 1179648. A maior lista válida (16384 chaves, 64 emissores revogados, datas com ano expandido, `seq = Number.MAX_SAFE_INTEGER`) tem 1102534 bytes. A relação com `MAX_INVENTORY` passa a ser `>=`. +- `packages/core/lib/blocklist-net.js`: `MAX_BLOCKLIST_MESSAGE` acompanha `MAX_SNAPSHOT_BYTES`. +- `apps/cli/lib/account/mod-routes.js`: `ACK_BODY_MAX` 524288 -> 1572864 (lista máxima mais 4096 ids de mudança e os campos assinados). +- `apps/cli/lib/mod-apply.js`: `PULL_BODY_CAP` 1048576 -> 1572864 (1000 mudanças mais a lista máxima). +- Prazos que crescem com os corpos maiores: a rota `/account/mod/api/ack` passa a ser lida como o anexo de uma denúncia (`apps/cli/lib/account/mod-routes.js`, leitura compartilhada `declaredLength` / `readLargeBody`): exige `Content-Length` (411 sem ele), responde 413 sem ler acima de `ACK_BODY_MAX`, corta um corpo parado por 5 s e dá 2 s mais o tamanho declarado a 1 Mbit/s (`largeBodyDeadlineMs`: cerca de 14,6 s para 1,5 MiB, antes 10 s fixos). Não há limite de leituras de `ack` ao mesmo tempo: um limite ocupado antes de conferir a assinatura deixaria qualquer pessoa segurar o `ack` do operador, então essas leituras contam só para o teto de 64 conexões do serviço. O `requestTimeout` do serviço passa a ser o tempo dos cabeçalhos mais o maior desses prazos (anexo ou ack, 18,6 s). No `mod apply`, `pull` e `ack` esperam até `TRANSFER_TIMEOUT_MS = 30000`; o desafio continua com 10 s. +- `packages/core/lib/mod-request.js`: `MAX_CHANGE_IDS` continua 4096. Ele limita os ids de um `ack`, não o tamanho da lista; o serviço entrega no máximo 1000 mudanças por `pull`. +- Sem mudança, conferidos: `apps/cli/lib/commands/mod.js` e `moderation-store.js` (`lotado`) leem `MAX_BLOCKED`; `readBlocklist`/`writeBlocklist` (core e web-member) não têm teto de tamanho; o nó do navegador recebe a lista pelo link do semeador, limitado por `MAX_EXTERNAL_FRAME` (8 MiB). +- `docs/protocol.md` seção 3 (com o parágrafo de compatibilidade), "Account deletion" e o mapa de módulos; `README.md` "Excluir conta" e os limites da moderação. + +## Compatibilidade (decisão da manutenção, 2026-11-17: assinar até 16384 já) + +- Até 4096 chaves: todas as versões leem a lista igual. +- De 4097 a 5796 chaves: a mensagem cabe no antigo teto de 393216 bytes (5796 é o pior caso: 64 emissores revogados, datas com ano expandido, `seq = Number.MAX_SAFE_INTEGER`; sem emissor revogado, com `seq = 1` e datas atuais, cabem 5861). Conferido com `createSnapshot` de verdade, medindo `JSON.stringify({ type: 'snapshot', snapshot })` para cada tamanho de lista. Um nó de versão anterior lê a lista como `'malformed'`, continua aplicando a última lista válida até ser atualizado e conclui a sincronização normalmente. +- Acima disso: um nó de versão anterior descarta a mensagem pelo tamanho, antes de ler, e conta uma mensagem inválida no canal `varanda/blocklist/1`. No nó do navegador de versão anterior (o app nativo, cujo worklet roda `packages/web-member`, e o PWA em cache), a meta G3 de `packages/web-member/lib/sync.js` (`blocklistSettled`) nunca se cumpre: todo Sincronizar termina em `tempo esgotado: blocklist`, e a sincronização contínua nunca informa que está pronta. Isso dura até o app ser atualizado. O nó do core de versão anterior não tem essa meta e não trava. +- Hoje a lista está muito abaixo de 4096 chaves. Quando a lista assinada se aproximar de 5796 chaves, o operador confere antes que as instalações já passaram da versão anterior (item na #224). + +## Testes (cada um falha com o valor antigo; conferido revertendo cada mudança) + +- `packages/core/test/blocklist.test.js`: a lista de 16384 chaves é criada, verificada e cabe em `MAX_SNAPSHOT_BYTES`; 16385 chaves são `'malformed'` e `createSnapshot` recusa; `MAX_BLOCKED >= MAX_INVENTORY`. Revertido: 3 asserções falham. +- `packages/core/test/blocklist-net.test.js`: uma lista de 16384 chaves (mais de 393216 bytes) atravessa o canal e é gravada sem mensagem inválida. Revertido: não chega. +- `apps/cli/test/account-mod.test.js`: um `ack` HTTP real com a lista de 16384 chaves responde 200, o espelho mostra 16384 bloqueios e um `pull` assinado devolve a lista. Revertido: a rota recusa o corpo. +- `apps/cli/test/mod-apply.test.js`: o `pull` de uma lista cheia (mais de 1048576 bytes) é aceito. Revertido: 4 asserções falham. +- `apps/cli/test/moderation-store.test.js`: o espelho de 16384 chaves reabre inteiro de `moderation.json`, e `lotado` vale em 16384. Revertido: falha. +- `packages/core/test/mod-request.test.js`: `MAX_CHANGE_IDS <= MAX_BLOCKED`. +- `apps/cli/test/account-mod.test.js`, prazos do `ack`: o prazo de um corpo máximo passa de 10 s e cresce com o tamanho, e o `requestTimeout` cobre o tempo dos cabeçalhos mais o maior prazo; um `ack` sem `Content-Length` recebe 411 com `Connection: close`; um `Content-Length` acima de `ACK_BODY_MAX` recebe 413 sem leitura; com três corpos anônimos de `ack` parados e abertos, um `ack` assinado de verdade responde 200; um corpo de 8000 bytes que pinga é cortado em cerca de 2 s. Revertido: o prazo sem a função falha, o 411 vira 403, o corte do pingo vem em 10 s. Com um limite de 2 leituras posto de volta só para conferir, o `ack` assinado recebe 503 `limite`. O 413 já valia antes e continua. +- `apps/cli/test/mod-apply.test.js`: `pull` e `ack` rodam com `TRANSFER_TIMEOUT_MS` (pelo menos o prazo máximo do `ack` no serviço mais 5 s), os desafios com 10 s. Revertido: falha. + +## Comandos + +```sh +npm run lint +node packages/core/test/blocklist.test.js +node packages/core/test/blocklist-net.test.js +node packages/core/test/mod-request.test.js +node packages/core/test/blocklist-member.test.js +node packages/core/test/blocklist-seeder.test.js +node apps/cli/test/account-mod.test.js +node apps/cli/test/moderation-store.test.js +node apps/cli/test/mod-apply.test.js +node apps/cli/test/mod.test.js +node apps/cli/test/moderation-e2e.test.js +node packages/web-member/test/blocklist.test.js +``` + +## Não rodado + +Nenhum aparelho: a mudança não toca tela. Uma lista real com mais de 4096 chaves não existe hoje; a compatibilidade com nós antigos foi conferida lendo `blocklist-net.js` e pelos testes do canal. diff --git a/packages/core/lib/blocklist-net.js b/packages/core/lib/blocklist-net.js index f73fb107..678301ac 100644 --- a/packages/core/lib/blocklist-net.js +++ b/packages/core/lib/blocklist-net.js @@ -58,8 +58,9 @@ const clock = require('./clock') const PROTOCOL = 'varanda/blocklist/1' // Same bound as MAX_SNAPSHOT_BYTES in blocklist.js (a valid snapshot of -// MAX_BLOCKED keys is far below it, so the message wrapper fits too). -const MAX_BLOCKLIST_MESSAGE = 393216 +// MAX_BLOCKED keys is about 75 KB below it, so the 31-byte message wrapper +// fits too). A literal, so this module does not load blocklist.js. +const MAX_BLOCKLIST_MESSAGE = 1179648 const MIN_SNAPSHOT_INTERVAL_MS = 10000 // incoming snapshot messages per connection const MAX_BAD_BLOCKLIST = 5 // the maxBad-th bad message drops the peer // Our own snapshot sends are spaced a quarter longer than the receiver's diff --git a/packages/core/lib/blocklist.js b/packages/core/lib/blocklist.js index 3328c76f..c3899a9c 100644 --- a/packages/core/lib/blocklist.js +++ b/packages/core/lib/blocklist.js @@ -17,13 +17,20 @@ const cert = require('./cert') const LF = '\n' const PREFIX = 'varanda-blocklist-v1' const SNAPSHOT_KEYS = ['v', 'seq', 'date', 'blocked', 'revokedIssuers', 'moderator', 'moderatorDate', 'moderatorSignature', 'signature'] -// Equal to MAX_INVENTORY (packages/core/lib/admission.js:18): a seeder never -// serves more logs than one snapshot can block. -const MAX_BLOCKED = 4096 +// At least MAX_INVENTORY (packages/core/lib/admission.js): a seeder never +// serves more logs than one snapshot can block. Above it because a deleted +// account's key stays blocked for good (certificates never expire), so the +// list only grows. Raised from 4096: an older build reads a longer list as +// 'malformed' and keeps its last valid one (docs/protocol.md section 3). +const MAX_BLOCKED = 16384 const MAX_REVOKED_ISSUERS = 64 -// 4096 x 67 bytes (64 hex, two quotes, a comma) + 64 x 67 + the header fits -// with room to spare, and stays far under MAX_EXTERNAL_FRAME (seeder.js:102). -const MAX_SNAPSHOT_BYTES = 393216 +// 16384 x 67 bytes (64 hex, two quotes, a comma) + 64 x 67 + the header is +// 1102534 bytes at most (expanded-year dates, seq MAX_SAFE_INTEGER): 1.125 +// MiB leaves about 75 KB of room and stays far under MAX_EXTERNAL_FRAME +// (frame-guard.js, 8 MiB). Every bound that carries a snapshot is sized +// from this one (blocklist-net.js, apps/cli mod-apply.js and +// account/mod-routes.js). +const MAX_SNAPSHOT_BYTES = 1179648 // A sorted list of distinct lowercase hex64 keys, at most `max` long. // Strictly increasing implies both sorted and distinct. diff --git a/packages/core/lib/mod-request.js b/packages/core/lib/mod-request.js index 5f2a3494..74673c60 100644 --- a/packages/core/lib/mod-request.js +++ b/packages/core/lib/mod-request.js @@ -30,8 +30,11 @@ const ORIGIN_RE = /^https:\/\/[a-z0-9.-]+(?::[0-9]{1,5})?$/ // Loopback over plain HTTP, for the operator's local tests only. const LOOPBACK_ORIGIN_RE = /^http:\/\/127\.0\.0\.1(?::[0-9]{1,5})?$/ const CHANGE_ID_RE = /^[A-Za-z0-9_-]{1,64}$/ -// Equal to MAX_BLOCKED (blocklist.js): one ack never names more changes than -// one snapshot can hold. +// At most MAX_BLOCKED (blocklist.js), so one ack never names more changes +// than one snapshot can hold, and at least MAX_PENDING (apps/cli +// account/moderation-store.js), the most changes one pull hands out. Not +// raised with MAX_BLOCKED: it also sizes the ack body (account/mod-routes.js +// ACK_BODY_MAX). const MAX_CHANGE_IDS = 4096 const HEX64 = /^[0-9a-f]{64}$/ const HEX128 = /^[0-9a-f]{128}$/ diff --git a/packages/core/test/blocklist-net.test.js b/packages/core/test/blocklist-net.test.js index cd30630f..31a89654 100644 --- a/packages/core/test/blocklist-net.test.js +++ b/packages/core/test/blocklist-net.test.js @@ -15,6 +15,7 @@ const NoiseSecretStream = require('@hyperswarm/secret-stream') const { attachAdmission } = require('../lib/admission') const { PROTOCOL, + MAX_BLOCKLIST_MESSAGE, MAX_RETRY_DELAY_MS, MESSAGE_BURST, MESSAGE_RATE, @@ -575,6 +576,25 @@ test('haves carry the signature prefix of the held snapshot; remoteHave keeps it t.is(h2.remoteHave, null) }) +test('a list of MAX_BLOCKED keys travels in one message (more than the old 4096)', async (t) => { + const bl = require('../lib/blocklist') + t.is(MAX_BLOCKLIST_MESSAGE, bl.MAX_SNAPSHOT_BYTES, 'same bound as MAX_SNAPSHOT_BYTES') + const blocked = Array.from({ length: bl.MAX_BLOCKED }, (_, i) => i.toString(16).padStart(64, '0')) + const big = snap(D1, 2, { blocked, revokedIssuers: [], signature: SIG_A + SIG_B + SIG_A + SIG_B }) + const bytes = b4a.byteLength(JSON.stringify({ type: 'snapshot', snapshot: big })) + t.ok(bytes > 393216, 'over the old bound: ' + bytes + ' bytes') + t.ok(bytes <= MAX_BLOCKLIST_MESSAGE, 'under the bound') + const { a, b } = streams(t) + const sa = fakeState(big) + const sb = fakeState(snap(D1, 1)) + const ha = attachBlocklist(a, { current: sa.current, offer: sa.offer }) + const hb = attachBlocklist(b, { current: sb.current, offer: sb.offer }) + await until(() => sb.saved.length === 1, { name: 'large snapshot persisted' }) + t.is(sb.saved[0].blocked.length, bl.MAX_BLOCKED) + t.is(ha.badMessages + hb.badMessages, 0) + t.absent(b.destroyed) +}) + test('close unpairs and settles', async (t) => { const { a, b } = streams(t) const sa = fakeState(null) diff --git a/packages/core/test/blocklist.test.js b/packages/core/test/blocklist.test.js index f8296aba..e9bbe3e1 100644 --- a/packages/core/test/blocklist.test.js +++ b/packages/core/test/blocklist.test.js @@ -163,9 +163,11 @@ test('verifySnapshot rejections', (t) => { t.ok(modCert) }) -test('size: 4096 blocked keys fit MAX_SNAPSHOT_BYTES, 4097 are malformed', (t) => { - t.is(bl.MAX_BLOCKED, 4096) - t.is(bl.MAX_SNAPSHOT_BYTES, 393216) +test('size: 16384 blocked keys fit MAX_SNAPSHOT_BYTES, 16385 are malformed', (t) => { + t.is(bl.MAX_BLOCKED, 16384) + t.is(bl.MAX_SNAPSHOT_BYTES, 1179648) + // A seeder never serves more logs than one snapshot can block. + t.ok(bl.MAX_BLOCKED >= require('../lib/admission').MAX_INVENTORY, 'MAX_BLOCKED >= MAX_INVENTORY') // the longest canonical dates (expanded years) const longDate = '+275760-09-13T00:00:00.000Z' t.ok(cert.isCanonicalIso(longDate)) @@ -179,6 +181,7 @@ test('size: 4096 blocked keys fit MAX_SNAPSHOT_BYTES, 4097 are malformed', (t) = const over = { ...s, blocked: blocked.concat([hexKey(bl.MAX_BLOCKED)]) } over.signature = b4a.toString(crypto.sign(bl.snapshotPayload(over), modKp.secretKey), 'hex') t.alike(bl.verifySnapshot(over, assocHex), { ok: false, reason: 'malformed' }) + t.exception.all(() => bl.createSnapshot({ seq: 1, date: longDate, blocked: over.blocked }, modKp, modCert), /blocked holds more than 16384 keys/) }) test('acceptSnapshot and compareSnapshots', (t) => { diff --git a/packages/core/test/mod-request.test.js b/packages/core/test/mod-request.test.js index f53de6c5..48bd5b2d 100644 --- a/packages/core/test/mod-request.test.js +++ b/packages/core/test/mod-request.test.js @@ -61,6 +61,7 @@ test('constants', (t) => { t.ok(Object.isFrozen(OPS)) t.alike([...SNAPSHOT_KEYS], blocklist.SNAPSHOT_KEYS, 'same keys and order as blocklist.js') t.is(MAX_CHANGE_IDS, 4096) + t.ok(MAX_CHANGE_IDS <= blocklist.MAX_BLOCKED, 'one ack never names more changes than one snapshot holds') t.ok(NONCE_RE.test('A-_9'.repeat(32))) t.absent(NONCE_RE.test('a'.repeat(42))) t.absent(NONCE_RE.test('a'.repeat(129)))