Recheck Downloader logic

[AskAlexSharov]

Artem reported that:
“I changed format existing files - i didn’t increment version. And seems downloader removed my new files and re-download old”.

Please re-check if it’s true. If yes - clearly understand “why” and “when”. For example: “if preverified.toml exists” or something else.

[JkLondon]

@AskAlexSharov

When: the filename is in the preverified registry and either the local .torrent is missing, or its infohash ≠ the preverified one.

Why: the downloader only trusts the preverified registry. On infohash mismatch it calls invalidateData (downloader.go:1076 → downloader.go:1004) — the local .seg is renamed to .part and re-downloaded.

There is a "keep existing local torrent" fallback (downloader.go:1047), but it only fires if the torrent is already loaded in the client. In erigon AddTorrentsFromDisk runs after the snapshot stage (backend.go:974), so the fallback misses. Same ordering issue as #19435.

Bottom line: changing format without bumping the version in the filename is unsafe — preverified always wins.

[AskAlexSharov]

"Bottom line: changing format without bumping the version in the filename is unsafe — preverified always wins." - aha.

1. How we can fail-fast assert it? To avoid "i spent week to produce new files and erigon removed all my work at startup"
2. Does it means we need db/snaptype: add cache-busting hash particle to snapshot filenames #19150 ?
3. Do you think - "changing format without bumping the version in the filename is unsafe — preverified always wins" - is right invariant?

[anacrolix]

This is an interesting case. There's some related issues that describe this problem: When should we leave the snapshots alone, and when should we clobber everything.

[sudeepdino008]

1. snapshotter should be run with --no-downloader-- in which cause this issue is moot. But ofc needs to be still solved.
2. "changing format without bumping the version in the filename is unsafe — preverified always wins" -- maybe we can decide for ourselves because user don't change the format. It's upto us what it should be.

We already have property -- if preverified.toml exists => no more downloading from preverified.toml
This scenario of verification + fix possible even after preverified.toml exists -- seems like some edge case

[anacrolix]

1. snapshotter should be run with --no-downloader-- in which cause this issue is moot. But ofc needs to be still solved.
2. "changing format without bumping the version in the filename is unsafe — preverified always wins" -- maybe we can decide for ourselves because user don't change the format. It's upto us what it should be.

We already have property -- if preverified.toml exists => no more downloading from preverified.toml This scenario of verification + fix possible even after preverified.toml exists -- seems like some edge case

I think you've nailed this. That's precisely the current semantics.

[AskAlexSharov]

user don't change the format - "files format reproducibility" is extremely hard feature:

• different OS, on different CPU, need to produce same file - even that we have several Compression algorithms, Concurrent-files-building by various amount of workers, Commitment dereferencing threshold, various --batchSize, etc...

Assume "that will never happen" - is wrong. (bugs/races/new_archs/etc...)

Also in Discord we can say for User: run erigon seg rm-state --latest (to do some hot-fix for future bug). In this case user's machine can produce file which it already has in preverified.toml. We can't use erigon seg rm-state --latest tool on user's machines? What will happen in this case (downloader will start mutate files at Erigon's runtime and Erigon will get segfault - from reading after end of file because end of file truncated)?

How about my dev machine? my laptop? If i want re-exec chiado from 0 to test new execution feature (no disk format change) - now i do it by erigon rm-all-state and then just start Erigon - should I do it with --no-downloader? Then what tooling is safe to use?

[AskAlexSharov]

Another usual way i exec Erigon from 0 is: start Erigon on empty datadir with --snap.skip-state-snapshot-download it downloading block files end exec blocks from 0.

[AskAlexSharov]

That's precisely the current semantics - semantic needs to be enforced somehow. If I will forget about this semantic - it's super easy for me to:

• "oh, i wanna see how many peers we have in Bittorrent net - will remove --no-downloader file"
• "oh, i added feature X to downloader - will remove --no-downloader flag on snapshotter - so i will have peer to test my new feature"
• "We starting support new chain - Arbitrum/Bloatnet - Artem (dev who usually not operating snapshotter servers) responsible for that chain - he will configure new snapshotter and release new files format there"
• etc...

[AskAlexSharov]

also today i played with dev server - where i tested feature which changing data format (i didn't increment version - because i just do experimentation - it's far from release files).
and after this i wanted switch back to main. i did erigon seg reset - and ofc it worked on "file names" instead of "file hashes" (it's known thing). But Erigon anyway didn't re-download "right" files (which are in preverified.toml). After several restarts/reset/go-to-chain-tip - i still see [WARN] [05-25|08:28:27.075] Preverified list reduced after applying type filter from=6686 to=6685
a bit unclear what to do