Repository navigation
feat(seeder): keep only what still counts of a cut writer - #732
Open
rodrigopavezi wants to merge 28 commits into
Open
rodrigopavezi wants to merge 28 commits into
rodrigopavezi wants to merge 28 commits into
Conversation
A copy counts up to its first hole, and past it only the leaves and key blocks a cut keeps, once the hole is at or past the cut. A key block citing a cut CW is judged by the CW's signed length, so a copy trimmed at the cut reads the history the whole copy reads (#350).
A node reads each group writer up to its first hole, as before, and past it only the leaves and key blocks it holds, so a cut writer a seeder keeps sparse folds like the whole copy (#350).
The browser node reads writers as the core node does: past the first hole, only the leaves and key blocks held. The fold memo stamps the blocks held past the contiguous length, and the community fold takes the diaries' cut marks, so a cut CW held sparse folds like the whole one (#350).
The account service reads the writers a seeder lists as the nodes do: past the first hole, the leaves and key blocks held, so a cut writer the seeder keeps sparse folds like the whole copy (#350).
A writer its diary cut keeps its blocks below the cut and, past it, only its leaves and key blocks: the rest is fetched a chunk at a time, read and cleared, and the quota counts what is kept. A released writer is a stub (block 0, leaves, key blocks), also a replaced boxless writer of a private group, which readers then reject by its header. The binding persists the cut and the range read before clearing; a restart recounts it, a restored cut clears nothing new until the scan confirms it, and a cut that is gone fetches again what clearing removed. A lower cut lifts a refused join once; a writer dropped for what it keeps is refused like one at admission (#350).
…pies A fresh member that syncs only with a seeder holds a frozen writer below its cut plus its leave, and a released writer as a stub, and folds the community and both groups exactly as after taking the whole copies from their owner (#350).
The protocol states what a seeder keeps of a cut or released writer, how readers read a writer past its first hole and how the fold counts it, and that a key block citing a cut CW is judged by the signed length (#350). Adds the changelog fragment and the verification file of the branch.
A foreign writer held up to a hole past what the seeder lists counts as caught up once every block a synced peer advertises below its signed length is held (core/lib/sparse-held.js). An own writer the diary cut is pushed once the seeder holds it up to the cut.
The browser node's goals follow the core node: a foreign writer held past the seeder's prefix is met once every advertised block is held, and an own writer the diary cut counts pushed at the cut.
A listed writer's fetch ends once every block the seeder advertises below its signed length is held, so a group with a writer the seeder keeps sparse no longer answers indisponivel to a page's join or post.
Completion goals in protocol.md now cover a writer held sparse, an own writer measured against its diary cut, and the account service's fetch. The verification file adds this round's tests, mutation proofs and open points; the changelog adds what a user sees.
… stub groupWalkCut gives 0 for a released writer and for one cut at 0 while still counted, and both are kept as stubs; the comments named only the first.
…s by bits A want asks for a whole bitfield segment and the answer carries its first page and every later page the peer has, so a page it holds nothing in never comes: a sparse peer on a log past 32768 blocks never counted as answered. readHeldBlocks now jumps past a hole to the next held block through the bitfield has() reads, instead of one has() per block.
…gain A chunk past a cut waited for every one of its blocks, so a seeder fed only by peers that keep the writer sparse never left the first chunk and never fetched a leave or key block beyond it. A chunk now ends once every block its synced peers advertise is held; what it holds is kept or cleared as before, and a chunk read that way joins the binding's unsure range (unsureFrom, unsureTo), read again when a peer joins and at growing intervals until a peer serves it whole.
… the cut The owner counted a writer its diary cut as pushed once a seeder held it up to the cut, so a leave or key block past the cut that never reached the seeder was confirmed and a once run ended ok with it on the device only. Both nodes now also wait until the seeder's replication peer advertises, or was served, each own group-leave and group-key block past the cut, and ask that peer for its bitfield there, since a full core sends no wants of its own.
A replica with more than 4096 blocks between its contiguous length and its signed length got no stamp, so a group with a long released stub or a writer cut early lost its fold memo for good. The stamp now names the runs of held blocks, read from the bitfield, and is null only past 4096 runs.
A tab checked a sparse writer's goal every 100 ms for as long as it was open, each tick reading every unmet goal. The wait now doubles up to 5 s while the held range stays the same, and a download or a peer coming or going brings it back to 100 ms. The sparse tests also describe the owner's push goal past the cut.
Protocol: a chunk past a cut ends at what its synced peers advertise and is read again later, a peer answers by bitfield segment, the owner waits for its own leaves and key blocks past the cut, the memo stamp names runs and the browser's sparse goal check backs off up to 5 s. The changelog and the verification file follow, with the third round's runs and mutation proofs.
A drop destroys the chunk's range, whose done() never settles, and wakes only a wait already armed, so one landing while the loop checked what its peers advertise parked it for good with a peer-add listener on a closed core. The loop now checks for a drop or a close after each await.
No test failed when peerHolds stopped counting the blocks this core served to the peer, which the core node's owner check can need.
… block there Every peer joining a cut writer reset the backoff to 1 s and read the whole unsure range again, so on a busy seeder each reader that connected, or a peer reconnecting in a loop, drove a full pass for a writer whose device is gone. A peer that joins now reads it again at once only when, once synced and asked for its bitfield there, it advertises a block in the range the seeder lacks; any other peer leaves the backoff.
Protocol: a peer that joins reads an unsure range again at once only when it advertises a block there the seeder lacks. The verification file adds the fourth round's changes, runs and mutation proofs, and names the single-block announcement among the answered approximation's causes.
# Conflicts: # docs/protocol.md # packages/core/lib/seeder.js # packages/core/lib/sync.js # packages/core/test/seeder-groups-cuts.test.js # packages/web-member/lib/member-ops.js # packages/web-member/lib/sync.js
A deleted group-cut still counts since #717, so a delete no longer makes a cut go: one test now checks that the deleted cut stays, after a restart too, and another drives the dropped-cut path from a stale admitted.json. The stored-binding unit test from #721 gets a scanned diary and a served entry, which _groupCutScan reads on every reconcile.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parte da #350.
O que muda e por quê
Depois do #713, o semeador guarda até 16 escritores por diário e grupo, mas guardava cada escritor cortado inteiro. Agora ele guarda só o que ainda conta.
bindCores).sync --oncetermina com um escritor guardado esparso quando todo bloco que os pares anunciam está guardado. O dono só dá o escritor cortado por enviado quando o semeador já tem as saídas e os blocos de chave dele depois do corte.indisponivel.docs/protocol.md("Cut writers", "Sparse copies", "Completion goals") descreve as regras. Formato, RPC e Termos não mudam.Como verificar
Os testes, o que cada um cobre, as mutações que falham sem cada correção e os riscos que ficam estão em
docs/verification/feat-writer-slot-sparse.md.Operador: o semeador novo vai antes dos apps.
Local, com a
mainem 4e901c1:npm run lintlimpo,npm run demotermina com DEMO OK enpm test -- --affected=origin/maincom 841 arquivos e uma falha, o tempo esgotado emapps/cli/test/update.test.jssob carga; sozinho (VARANDA_TEST_CONCURRENCY=1), ele passa com 73/73 testes. Depois veio amainem fcaecba, num merge sem conflito.Checklist
npm run lint,npm test -- --affected=origin/mainenpm run demopassamdocs/changelog/feat-writer-slot-sparse.mde verificação emdocs/verification/feat-writer-slot-sparse.md