scalar reports network errors and retries over and over when cache is corrupt

[mbest]

I noticed when running git fetch that I was getting network errors. I then tried running scalar run all and it was doing the same. I did a bunch of digging to try to find the problem and finally found that there were files in my scalar cache that were blank. scalar run all actually flagged these files specifically but reported the problem as a network error. Here are some logs:

> git fetch
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) (bytes received): 17086866, done.
Installing prefetch packfiles:  33% (1/3), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 1/6] (bytes received): 17086866, done.
Installing prefetch packfiles:  33% (1/3), done.
Waiting to retry after network error (sec): 100% (8/8), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 2/6] (bytes received): 17086866, done.
Installing prefetch packfiles:  33% (1/3), done.
Waiting to retry after network error (sec): 100% (16/16), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 3/6] (bytes received): 17086866, done.
Installing prefetch packfiles:  33% (1/3), done.
Waiting to retry after network error (sec): 100% (32/32), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 4/6] (bytes received):
  17086866, done.
Installing prefetch packfiles:  33% (1/3), done.
Waiting to retry after network error (sec): 100% (64/64), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 5/6] (bytes received): 17086866, done.
Installing prefetch packfiles:  33% (1/3), done.
Waiting to retry after network error (sec): 100% (128/128), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 6/6] (bytes received): 27119592, done.
Installing prefetch packfiles:  25% (1/4), done.
Waiting to retry after network error (sec): 100% (256/256), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) (bytes received): 134005584, done.
Installing prefetch packfiles: 100% (1/1), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 1/6] (bytes received):
  134005584, done.
Installing prefetch packfiles: 100% (1/1), done.
Waiting to retry after network error (sec): 100% (8/8), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 2/6] (bytes received):
  133487121, done.
Installing prefetch packfiles: 100% (1/1), done.
Waiting to retry after network error (sec): 100% (16/16), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 3/6] (bytes received):
  134011915, done.
Installing prefetch packfiles: 100% (1/1), done.
Waiting to retry after network error (sec): 100% (32/32), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 4/6] (bytes received):
  134009342, done.
Installing prefetch packfiles: 100% (1/1), done.
Waiting on soft throttle (sec): 100% (159/159), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 5/6] (bytes received):
  134012239, done.
Installing prefetch packfiles: 100% (1/1), done.
Waiting on soft throttle (sec): 100% (124/124), done.
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) [retry 6/6] (bytes received):
  134012196, done.
Installing prefetch packfiles: 100% (1/1), done.
error: gvfs-helper error: 'index-pack failed on 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/pack/tempPacks/t-20260105-231120-703339-0027.pack''

> scalar run all
D:/1JS
error: inflate: data stream error (unknown compression method)
error: unable to unpack ebcfc2974f82b1adc6611f941938ccb54277f025 header
error: files 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/pack/tempPacks/t-20260106-002432-889450-0000.temp' and 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/eb/cfc2974f82b1adc6611f941938ccb54277f025' differ in contents
error: gvfs-helper error: 'could not install loose object 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/eb/cfc2974f82b1adc6611f941938ccb54277f025': from GET ebcfc2974f82b1adc6611f941938ccb54277f025'
Receiving packfile 1/1 with 1 objects (bytes received): 17367934, done.
Receiving packfile 1/1 with 1 objects [retry 1/6] (bytes received): 17367934, done.
Waiting to retry after network error (sec): 100% (8/8), done.
Receiving packfile 1/1 with 1 objects [retry 2/6] (bytes received): 17367934, done.
Waiting to retry after network error (sec): 100% (16/16), done.
Receiving packfile 1/1 with 1 objects [retry 3/6] (bytes received): 17367934, done.
...

Once I figured out how to work around this, I kept re-running scalar run all, killing it after the error and deleting the corrupt file it mentioned:

PS D:\1JS> scalar run all
D:/1JS
error: inflate: data stream error (unknown compression method)
error: unable to unpack f5e6eaab7e0aacb5f7bd866cf4c6becbf9dba1be header
error: files 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/pack/tempPacks/t-20260106-014459-496583-0001.temp' and 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/f5/e6eaab7e0aacb5f7bd866cf4c6becbf9dba1be' differ in contents
error: gvfs-helper error: 'could not install loose object 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/f5/e6eaab7e0aacb5f7bd866cf4c6becbf9dba1be': from GET f5e6eaab7e0aacb5f7bd866cf4c6becbf9dba1be'
PS D:\1JS> scalar run all
D:/1JS
error: inflate: data stream error (unknown compression method)
error: unable to unpack a163b1302d4729ebdb0a12d3876ca5bca4e1a8c3 header
error: files 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/pack/tempPacks/t-20260106-014520-049919-0001.temp' and 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/a1/63b1302d4729ebdb0a12d3876ca5bca4e1a8c3' differ in contents
error: gvfs-helper error: 'could not install loose object 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/a1/63b1302d4729ebdb0a12d3876ca5bca4e1a8c3': from GET a163b1302d4729ebdb0a12d3876ca5bca4e1a8c3'
PS D:\1JS> scalar run all
D:/1JS
error: inflate: data stream error (unknown compression method)
error: unable to unpack 5400a87518c5b04d27931149b24c6bc4cbaffe9d header
error: files 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/pack/tempPacks/t-20260106-014541-226688-0001.temp' and 'D:/.scalarCache/id_49b0c9f4-555f-4624-8157-a57e6df513b3/54/00a87518c5b04d27931149b24c6bc4cbaffe9d' differ in contents
PS D:\1JS> scalar run all
D:/1JS
Prefetch 1767158820 (2025-12-31 05:27:00 +0000) (bytes received): 17086866, done.
Installing prefetch packfiles: 100% (3/3), done.
From https://office.visualstudio.com/DefaultCollection/Office/_git/1JS
   270eb155883e..383ba624de5f  main       -> refs/prefetch/remotes/origin/main
warning: fetch normally indicates which branches had a forced update,
but that check has been disabled; to re-enable, use '--show-forced-updates'
flag or run 'git config fetch.showForcedUpdates true'
Enumerating objects: 275, done.
Counting objects: 100% (275/275), done.
Delta compression using up to 24 threads
Compressing objects: 100% (275/275), done.
Writing objects: 100% (275/275), done.
Total 275 (delta 39), reused 0 (delta 0), pack-reused 0 (from 0)
Enumerating objects: 8000, done.
Counting objects: 100% (8000/8000), done.
Delta compression using up to 24 threads
Compressing objects: 100% (7907/7907), done.
Writing objects: 100% (8000/8000), done.
Total 8000 (delta 43), reused 7957 (delta 0), pack-reused 0 (from 0)

[derrickstolee]

Thanks for the report. I've got a PR open for review to help with these error cases. I also saw some internal reports of similar issues so included advice related to those workarounds.