Skip to content

feat(seeder): keep only what still counts of a cut writer - #732

Open
rodrigopavezi wants to merge 28 commits into
mainfrom
feat/writer-slot-sparse
Open

rodrigopavezi wants to merge 28 commits into
mainfrom
feat/writer-slot-sparse

Conversation

@rodrigopavezi

Copy link
Copy Markdown
Contributor

Parte da #350.

O que muda e por quê

Depois do #713, o semeador guarda até 16 escritores por diário e grupo, mas guardava cada escritor cortado inteiro. Agora ele guarda só o que ainda conta.

  • De um escritor cortado ficam os blocos abaixo do corte e, a partir dele, só as saídas do grupo e os blocos de chave. De um escritor liberado (corte 0) fica um toco: o bloco 0, as saídas e os blocos de chave. A cota conta só o que fica guardado.
  • O nó core, o nó do navegador e o serviço de conta leem um escritor além do primeiro buraco (só as saídas e os blocos de chave), então quem só fala com semeadores dobra o grupo como quem tem as cópias inteiras. A regra do buraco mora na dobra (bindCores).
  • O sync --once termina com um escritor guardado esparso quando todo bloco que os pares anunciam está guardado. O dono só dá o escritor cortado por enviado quando o semeador já tem as saídas e os blocos de chave dele depois do corte.
  • Um semeador que só recebe o escritor de pares que o guardam esparso relê a faixa incerta: quando entra um par que anuncia ali um bloco que falta, e a cada até 5 minutos.
  • Entrar, postar ou selar pela Página num grupo com escritor esparso não responde mais indisponivel.

docs/protocol.md ("Cut writers", "Sparse copies", "Completion goals") descreve as regras. Formato, RPC e Termos não mudam.

Como verificar

Os testes, o que cada um cobre, as mutações que falham sem cada correção e os riscos que ficam estão em docs/verification/feat-writer-slot-sparse.md.

node packages/core/test/sparse-held.test.js
node packages/core/test/seeder-groups-cuts.test.js
node packages/core/test/sync-groups-sparse.test.js
node packages/core/test/seeder-delete.test.js
node packages/web-member/test/sync-groups-sparse.test.js
node packages/web-member/test/group-memo.test.js
node apps/cli/test/pages-groups-sparse.test.js

Operador: o semeador novo vai antes dos apps.

Local, com a main em 4e901c1: npm run lint limpo, npm run demo termina com DEMO OK e npm test -- --affected=origin/main com 841 arquivos e uma falha, o tempo esgotado em apps/cli/test/update.test.js sob carga; sozinho (VARANDA_TEST_CONCURRENCY=1), ele passa com 73/73 testes. Depois veio a main em fcaecba, num merge sem conflito.

Checklist

  • npm run lint, npm test -- --affected=origin/main e npm run demo passam
  • docs/protocol.md atualizado (formato e RPC não mudam)
  • Entrada do changelog em docs/changelog/feat-writer-slot-sparse.md e verificação em docs/verification/feat-writer-slot-sparse.md
  • Textos para o usuário estão em pt-BR (nenhum texto novo)
  • Conferido no Android e no simulador iOS: não se aplica, a mudança é do nó, do semeador e do serviço de conta, sem tela
  • Nenhum segredo, chave ou dado pessoal no diff

A copy counts up to its first hole, and past it only the leaves and key
blocks a cut keeps, once the hole is at or past the cut. A key block
citing a cut CW is judged by the CW's signed length, so a copy trimmed
at the cut reads the history the whole copy reads (#350).
A node reads each group writer up to its first hole, as before, and past
it only the leaves and key blocks it holds, so a cut writer a seeder
keeps sparse folds like the whole copy (#350).
The browser node reads writers as the core node does: past the first
hole, only the leaves and key blocks held. The fold memo stamps the
blocks held past the contiguous length, and the community fold takes the
diaries' cut marks, so a cut CW held sparse folds like the whole one
(#350).
The account service reads the writers a seeder lists as the nodes do:
past the first hole, the leaves and key blocks held, so a cut writer the
seeder keeps sparse folds like the whole copy (#350).
A writer its diary cut keeps its blocks below the cut and, past it, only
its leaves and key blocks: the rest is fetched a chunk at a time, read and
cleared, and the quota counts what is kept. A released writer is a stub
(block 0, leaves, key blocks), also a replaced boxless writer of a private
group, which readers then reject by its header.

The binding persists the cut and the range read before clearing; a
restart recounts it, a restored cut clears nothing new until the scan
confirms it, and a cut that is gone fetches again what clearing removed.
A lower cut lifts a refused join once; a writer dropped for what it keeps
is refused like one at admission (#350).
…pies

A fresh member that syncs only with a seeder holds a frozen writer below
its cut plus its leave, and a released writer as a stub, and folds the
community and both groups exactly as after taking the whole copies from
their owner (#350).
The protocol states what a seeder keeps of a cut or released writer, how
readers read a writer past its first hole and how the fold counts it, and
that a key block citing a cut CW is judged by the signed length (#350).
Adds the changelog fragment and the verification file of the branch.
A foreign writer held up to a hole past what the seeder lists counts as
caught up once every block a synced peer advertises below its signed
length is held (core/lib/sparse-held.js). An own writer the diary cut is
pushed once the seeder holds it up to the cut.
The browser node's goals follow the core node: a foreign writer held past
the seeder's prefix is met once every advertised block is held, and an own
writer the diary cut counts pushed at the cut.
A listed writer's fetch ends once every block the seeder advertises below
its signed length is held, so a group with a writer the seeder keeps
sparse no longer answers indisponivel to a page's join or post.
Completion goals in protocol.md now cover a writer held sparse, an own
writer measured against its diary cut, and the account service's fetch.
The verification file adds this round's tests, mutation proofs and open
points; the changelog adds what a user sees.
… stub

groupWalkCut gives 0 for a released writer and for one cut at 0 while
still counted, and both are kept as stubs; the comments named only the
first.
…s by bits

A want asks for a whole bitfield segment and the answer carries its first
page and every later page the peer has, so a page it holds nothing in
never comes: a sparse peer on a log past 32768 blocks never counted as
answered. readHeldBlocks now jumps past a hole to the next held block
through the bitfield has() reads, instead of one has() per block.
…gain

A chunk past a cut waited for every one of its blocks, so a seeder fed
only by peers that keep the writer sparse never left the first chunk and
never fetched a leave or key block beyond it. A chunk now ends once every
block its synced peers advertise is held; what it holds is kept or
cleared as before, and a chunk read that way joins the binding's unsure
range (unsureFrom, unsureTo), read again when a peer joins and at growing
intervals until a peer serves it whole.
… the cut

The owner counted a writer its diary cut as pushed once a seeder held it
up to the cut, so a leave or key block past the cut that never reached
the seeder was confirmed and a once run ended ok with it on the device
only. Both nodes now also wait until the seeder's replication peer
advertises, or was served, each own group-leave and group-key block past
the cut, and ask that peer for its bitfield there, since a full core
sends no wants of its own.
A replica with more than 4096 blocks between its contiguous length and
its signed length got no stamp, so a group with a long released stub or
a writer cut early lost its fold memo for good. The stamp now names the
runs of held blocks, read from the bitfield, and is null only past 4096
runs.
A tab checked a sparse writer's goal every 100 ms for as long as it was
open, each tick reading every unmet goal. The wait now doubles up to 5 s
while the held range stays the same, and a download or a peer coming or
going brings it back to 100 ms. The sparse tests also describe the
owner's push goal past the cut.
Protocol: a chunk past a cut ends at what its synced peers advertise and
is read again later, a peer answers by bitfield segment, the owner waits
for its own leaves and key blocks past the cut, the memo stamp names
runs and the browser's sparse goal check backs off up to 5 s. The
changelog and the verification file follow, with the third round's runs
and mutation proofs.
A drop destroys the chunk's range, whose done() never settles, and wakes
only a wait already armed, so one landing while the loop checked what
its peers advertise parked it for good with a peer-add listener on a
closed core. The loop now checks for a drop or a close after each await.
No test failed when peerHolds stopped counting the blocks this core
served to the peer, which the core node's owner check can need.
… block there

Every peer joining a cut writer reset the backoff to 1 s and read the
whole unsure range again, so on a busy seeder each reader that connected,
or a peer reconnecting in a loop, drove a full pass for a writer whose
device is gone. A peer that joins now reads it again at once only when,
once synced and asked for its bitfield there, it advertises a block in
the range the seeder lacks; any other peer leaves the backoff.
Protocol: a peer that joins reads an unsure range again at once only when
it advertises a block there the seeder lacks. The verification file adds
the fourth round's changes, runs and mutation proofs, and names the
single-block announcement among the answered approximation's causes.
# Conflicts:
#	docs/protocol.md
#	packages/core/lib/seeder.js
#	packages/core/lib/sync.js
#	packages/core/test/seeder-groups-cuts.test.js
#	packages/web-member/lib/member-ops.js
#	packages/web-member/lib/sync.js
A deleted group-cut still counts since #717, so a delete no longer makes
a cut go: one test now checks that the deleted cut stays, after a restart
too, and another drives the dropped-cut path from a stale admitted.json.
The stored-binding unit test from #721 gets a scanned diary and a served
entry, which _groupCutScan reads on every reconcile.
@rodrigopavezi
rodrigopavezi requested a review from a team as a code owner November 17, 2026 08:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant