Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
d25c480
fix(groups): fold a cut writer held sparse like the whole copy
rodrigopavezi Nov 17, 2026
833bc13
feat(core): read a group writer past its first hole
rodrigopavezi Nov 17, 2026
b457aa1
feat(web-member): read a group writer past its first hole
rodrigopavezi Nov 17, 2026
b546e77
feat(cli): read a listed group writer past its first hole
rodrigopavezi Nov 17, 2026
67b7998
feat(seeder): keep a cut writer sparse and a released one as a stub
rodrigopavezi Nov 17, 2026
ef5df14
test(core): fold sparse cut writers fed by a seeder like the whole co…
rodrigopavezi Nov 17, 2026
d946028
docs: describe sparse cut writers on seeders and readers
rodrigopavezi Nov 17, 2026
8ba3364
Merge remote-tracking branch 'origin/main' into feat/writer-slot-sparse
rodrigopavezi Nov 17, 2026
7ab4f7b
feat(sync): end a once run with a sparse cut writer
rodrigopavezi Nov 17, 2026
1a7148b
feat(web-member): end a once run with a sparse cut writer
rodrigopavezi Nov 17, 2026
895371f
feat(account): count a sparse cut writer whole in a page's group fold
rodrigopavezi Nov 17, 2026
d487e5e
docs: describe completion goals with sparse cut writers
rodrigopavezi Nov 17, 2026
ea60883
docs(seeder): name a writer cut at 0 once the releases are spent as a…
rodrigopavezi Nov 17, 2026
00bbc48
fix(sparse): count a peer answered by its bitfield segment, skip hole…
rodrigopavezi Nov 17, 2026
af985d3
fix(seeder): end a chunk past a cut fed by sparse peers and read it a…
rodrigopavezi Nov 17, 2026
f9c3b6c
fix(sync): confirm a cut writer once the seeder holds its leaves past…
rodrigopavezi Nov 17, 2026
cdd9947
fix(web): stamp the runs of blocks a replica holds past its prefix
rodrigopavezi Nov 17, 2026
18d4c3f
perf(web): back off the sparse goal check while nothing changes
rodrigopavezi Nov 17, 2026
9b2137f
docs: describe the third round of sparse cut writer fixes
rodrigopavezi Nov 17, 2026
4886120
fix(seeder): end the wait for a chunk past a cut on a drop or a close
rodrigopavezi Nov 17, 2026
e483d6e
test(sparse): cover a block served to the peer in peerHolds
rodrigopavezi Nov 17, 2026
6cad413
fix(seeder): read an unsure range again only for a peer that offers a…
rodrigopavezi Nov 17, 2026
95b4aa8
docs: describe the peer-offer reread and the fourth round's proofs
rodrigopavezi Nov 17, 2026
955ae56
Merge remote-tracking branch 'origin/main' into feat/writer-slot-sparse
rodrigopavezi Nov 17, 2026
7f6b972
test(seeder): fit the cut tests to the merge of main
rodrigopavezi Nov 17, 2026
0cc4535
fix(seeder): try a refused join again when a deleted cut arrives late
rodrigopavezi Nov 17, 2026
25fba70
docs: describe the merge of main and the retired writer cut
rodrigopavezi Nov 17, 2026
d7a7b38
Merge remote-tracking branch 'origin/main' into feat/writer-slot-sparse
rodrigopavezi Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 31 additions & 7 deletions apps/cli/lib/account/pages-groups.js
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,13 @@ const hcrypto = require('hypercore-crypto')
const Hypercore = require('hypercore')
const Corestore = require('corestore')
const { WebSocket, createWebSocketStream } = require('ws')
const { writeJsonAtomic, appendRecord, readAll, readBlocks, openReplica, purgeReplica } = require('@varanda/core/lib/store')
const { writeJsonAtomic, appendRecord, readAll, readBlocks, readHeldBlocks, openReplica, purgeReplica } = require('@varanda/core/lib/store')
const { refuseSymlinkDir, certForKey, seederCertForKey, isPlainObject } = require('@varanda/core/lib/cert')
const { attachAdmission } = require('@varanda/core/lib/admission')
const { attachGroups, GROUPS_LIMIT_MAX } = require('@varanda/core/lib/groups-net')
const { generateNoiseKeyPair } = require('@varanda/core/lib/net')
const { groupJoinRecord, latestProfile, validateRecord, MAX_LOG_BLOCKS } = require('@varanda/core/lib/records')
const { advertisedHeld } = require('@varanda/core/lib/sparse-held')
const { latestUsername } = require('@varanda/core/lib/username')
const groups = require('@varanda/core/lib/group')
const { keyTag, keyCommit, wipe } = require('@varanda/core/lib/group-crypto')
Expand All @@ -67,6 +68,9 @@ const NOISE_FILE = 'noise-key.json'
const MAX_REPLICAS = 1024
const MAX_WRITER_PAGES = Math.ceil(groups.MAX_GROUP_WRITERS / GROUPS_LIMIT_MAX)
const FETCH_TIMEOUT_MS = 10000
// A listed writer held past the seeder's prefix is checked this often for
// the blocks the seeder advertises (core sparse-held.js).
const SPARSE_POLL_MS = 50
const BACKOFF_MIN_MS = 1000
const BACKOFF_MAX_MS = 60000
const HANDSHAKE_TIMEOUT_MS = 10000
Expand Down Expand Up @@ -604,16 +608,34 @@ function createPagesGroups ({ dir, writer, pagesStore, keys, association, blockl

// The core at the length the seeder offers, held (release it), or null
// when nothing arrived in time. `complete` is false while a block up to
// min(length, MAX_LOG_BLOCKS) is missing.
async function fetch (key, held) {
// min(length, MAX_LOG_BLOCKS) is missing, or for a `sparse` core (a listed
// writer, which a seeder keeps sparse once its diary cut it, #350) while
// a block the seeder advertises below that is missing.
async function fetch (key, held, sparse = false) {
const core = await hold(key)
held.push(key)
await timeout(core.update({ wait: true }).catch(noop), fetchTimeoutMs)
if (core.length === 0) return null
const end = Math.min(core.length, MAX_LOG_BLOCKS)
await timeout(core.download({ start: 0, end }).done().catch(noop), fetchTimeoutMs)
const range = core.download({ start: 0, end }).done().catch(noop)
if (!sparse) {
await timeout(range, fetchTimeoutMs)
if (core.contiguousLength === 0) return null
return { core, complete: core.contiguousLength >= end }
}
// The range never completes past a seeder's hole: it ends once every
// block the seeder advertises is here.
const deadline = Date.now() + fetchTimeoutMs
let done = false
range.then(() => { done = true })
let complete = false
for (;;) {
complete = core.contiguousLength >= end || (core.contiguousLength > 0 && await advertisedHeld(core, end))
if (complete || done || closed || Date.now() >= deadline) break
await timeout(range, Math.min(SPARSE_POLL_MS, Math.max(0, deadline - Date.now())))
}
if (core.contiguousLength === 0) return null
return { core, complete: core.contiguousLength >= end }
return { core, complete }
}

function localWriter (key) {
Expand Down Expand Up @@ -699,7 +721,7 @@ function createPagesGroups ({ dir, writer, pagesStore, keys, association, blockl
// replica of the same key would never get a channel, so read it here.
let got = localWriter(it.key)
try {
if (got === null) got = await fetch(it.key, held)
if (got === null) got = await fetch(it.key, held, true)
} catch {
incomplete = true
continue
Expand All @@ -711,7 +733,9 @@ function createPagesGroups ({ dir, writer, pagesStore, keys, association, blockl
if (!got.complete) incomplete = true
const core = got.core
if (core.fork !== 0) continue
const entries = await readBlocks(core)
// A seeder keeps a cut writer sparse (#350): past its first hole, its
// leaves and key blocks.
const entries = await readHeldBlocks(core)
const h = entries[0] && entries[0].record
if (groups.validateWriterHeader(h) !== null || h.group !== key) continue
logs.push({ key: it.key, diary: h.diary, entries, length: core.contiguousLength, signed: core.length })
Expand Down
138 changes: 138 additions & 0 deletions apps/cli/test/pages-groups-sparse.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
'use strict'

// apps/cli/lib/account/pages-groups.js with a writer a seeder keeps sparse
// (#350, docs/protocol.md "Sparse copies"): its blocks below the diary cut,
// then only its leaves and key blocks. The page's fold counts such a copy
// whole once every block the seeder advertises is held. Its own file: the
// pages-groups.test.js runs already take most of the runner's per-file time.

const test = require('brittle')
const b4a = require('b4a')
const crypto = require('hypercore-crypto')
const { testnet, tmpDir, until } = require('./helpers/net')
const { createIssuerCert, createIssuedCert, createUsernameRecord, generateAssociationKeyPair } = require('@varanda/core/lib/cert')
const { signRoster, issuerCountersign } = require('@varanda/core/lib/page')
const member = require('@varanda/core/lib/member')
const coreStore = require('@varanda/core/lib/store')
const { groupPost, groupLeave } = require('@varanda/core/lib/group')
const { groupCutRecord } = require('@varanda/core/lib/records')
const { person, exchange, startSeeder, syncOnce, ownWriter, runUntil } = require('../../../packages/core/test/helpers/sync-groups')
const { offlineAppend, writerEntries } = require('../../../packages/core/test/helpers/group-member')
const { startSeedWs } = require('../lib/seed-ws')
const { openPageKeys } = require('../lib/account/page-keys')
const { createPagesWriter } = require('../lib/account/pages-writer')
const { openPagesStore } = require('../lib/account/pages-store')
const { createPagesGroups } = require('../lib/account/pages-groups')

const PAGE_KEYS = 'cd'.repeat(32)

function hex (buf) {
return b4a.toString(buf, 'hex')
}

let nonceCount = 0
function nonce () {
nonceCount++
return (String(nonceCount) + 'g'.repeat(43)).slice(0, 43)
}

function newPage (ctx, keys, username) {
const { key, row } = keys.pending(nonce())
keys.put(key, row)
const cert = createIssuedCert({ member: key }, ctx.issuerKp, ctx.issuerCert)
const usernameRecord = createUsernameRecord({ member: key, username }, ctx.issuerKp, ctx.issuerCert)
const owner = crypto.keyPair()
const fields = { page: key, gen: 1, owner: hex(owner.publicKey), admins: [], prev: null, at: new Date().toISOString(), nonce: nonce(), association: ctx.association }
const signature = signRoster(fields, owner)
const rosterRecord = issuerCountersign({ type: 'page-roster', ...fields, signature }, ctx.issuerKp, ctx.issuerCert)
return { key, owner: hex(owner.publicKey), args: { cert, usernameRecord, rosterRecord } }
}

// Retries while the fold is still partial ('indisponivel').
function settled (fn, name, timeoutMs) {
return until(async () => {
try {
return await fn()
} catch (err) {
if (err.code === 'indisponivel') return null
throw err
}
}, { name, timeoutMs, intervalMs: 500 })
}

async function blocksHeld (core, from, to) {
const out = []
for (let i = from; i < to; i++) if (await core.has(i)) out.push(i)
return out
}

test('pages-groups: a group with a writer the seeder keeps sparse is no longer indisponivel (#350)', async (t) => {
t.timeout(300000)
const { bootstrap } = await testnet(t)
const kp = generateAssociationKeyPair()
const association = hex(kp.publicKey)
const issuerKp = crypto.keyPair()
const ctx = { association, issuerKp, issuerCert: createIssuerCert({ issuer: hex(issuerKp.publicKey) }, kp) }
const seeder = await startSeeder(t, bootstrap, kp)
const server = await startSeedWs({ port: 0, seeder })
t.teardown(() => server.close(), { force: true })
const seederWs = 'http://127.0.0.1:' + server.port

const a = await person(t, kp, 'Ana')
const b = await person(t, kp, 'Bia')
const { community: C } = await member.communityCreate(a, { name: 'Bairro', join: 'open' })
const { group: G } = await member.groupCreate(a, { name: 'Praça', join: 'open', community: C })
await member.groupPost(a, { group: G, text: 'oi' })
await exchange(a, b)
await member.communityJoin(b, { community: C })
await member.groupJoin(b, { group: G })
await member.groupPost(b, { group: G, text: 'antes' })
// W, B's writer in G, is frozen at its length: a post and a leave past
// the cut.
const w = ownWriter(b, G)
const cut = (await writerEntries(b, G)).length
await offlineAppend(b, G, groupPost({ text: 'depois' }))
const leave = await offlineAppend(b, G, groupLeave({ seen: null, cuts: [{ writer: w, seq: cut + 1 }] }))
const s = await coreStore.openStore(b.dataDir)
try {
await coreStore.appendRecord(await coreStore.openDiary(s), groupCutRecord({ group: G, writer: w, cut }))
} finally {
await s.close()
}
t.ok((await syncOnce(bootstrap, a)).result.ok, 'A pushes its diary and roots')
await runUntil(b, bootstrap, async () => {
const e = seeder._groupServed.get(w)
const st = seeder._groups.get(w)
if (!e || !st || e.cutting || st.cut !== cut || st.clearedTo !== leave.seq + 1) return false
return (await blocksHeld(e.core, 0, leave.seq + 1)).length === cut + 1
})
t.alike(await blocksHeld(seeder._groupServed.get(w).core, 0, leave.seq + 1), [...Array(cut).keys(), leave.seq], 'the seeder keeps W below its cut and its leave')

const dir = tmpDir(t, 'varanda-account-')
const keys = openPageKeys({ dir, env: { VARANDA_PAGE_KEYS: PAGE_KEYS } })
const pagesStore = openPagesStore(dir)
const writer = createPagesWriter({ dir, keys, association, seederWs })
t.teardown(() => writer.close(), { force: true })
await writer.ready()
const pg = createPagesGroups({ dir, writer, pagesStore, keys, association, seederWs })
t.teardown(() => pg.close(), { force: true })
await pg.ready()
await until(() => writer.connected() && pg.connected(), { name: 'links up', timeoutMs: 20000 })
const page = newPage(ctx, keys, 'padaria')
await writer.createPage(page.args)
await pagesStore.create({ page: page.key, owner: page.owner })

t.is((await settled(() => pg.communityJoin(page.key, C), 'community join', 60000)).state, 'member', 'the page joins the community')
// Each try fetches W again; before #350's goal it waited out the fetch
// timeout and answered indisponivel every time.
t.is((await settled(() => pg.groupJoin(page.key, G), 'group join', 60000)).state, 'member', 'the page joins the group with W sparse')
// The seeder indexes the page's new writer first: its listing is partial
// meanwhile.
const posted = await settled(() => pg.groupPost(page.key, G, { text: 'pão quente' }), 'group post', 60000)
t.ok(Number.isSafeInteger(posted.seq), 'and posts')
const read = await pg.groupRead(page.key, G)
t.is(read.incomplete, false, 'its fold is whole')
const texts = read.items.map((p) => p.text)
t.ok(texts.includes('antes'), 'the post below the cut counts')
t.absent(texts.includes('depois'), 'the post past the cut does not')
})
3 changes: 3 additions & 0 deletions docs/changelog/feat-writer-slot-sparse.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Alterado

- Os semeadores passam a guardar só o que conta de um escritor cortado pelo diário (PR `feat/writer-slot-sparse`, parte da #350): os blocos abaixo do corte e, a partir dele, só as saídas do grupo e os blocos de chave. De um escritor liberado (corte 0) fica um toco: o bloco 0, as saídas e os blocos de chave. A cota conta o que fica guardado. O nó core, o nó do navegador e o serviço de contas leem um escritor além do primeiro buraco (só as saídas e os blocos de chave), então quem só fala com semeadores vê o grupo como quem tem as cópias inteiras. Um bloco de chave que cita um escritor de comunidade cortado vale pelo comprimento que o escritor assinou. O escritor sem caixa trocado num grupo privado, depois de liberado, também fica como toco no semeador. O `sync --once` (nó core e navegador) termina com um escritor guardado esparso quando todo bloco que os pares anunciam está guardado, e o dono dá o escritor cortado por enviado quando o semeador o guarda até o corte e já tem as saídas e os blocos de chave dele depois do corte. Um semeador que só recebe o escritor de pares que o guardam esparso também chega às saídas depois do corte. Entrar, postar ou selar pela Página num grupo com escritor esparso não responde mais `indisponivel`. Conferência em `docs/verification/feat-writer-slot-sparse.md`.
Loading
Loading