Finished Jobs waiting in `Running` state while another job is in `Generating Report` state preventing other jobs from starting

[Skgland]

Ideally finished jobs would transition into the NeedsReport state upon completion to make room for the next job to start.

Currently

1. Only one job can be in the GeneratingReport state

2. Only one job can be in the running state

3. Running jobs transition directly to GeneratingReport once completed and no job is already generating a report.

4. and 2. are reasonable and fine, but I think 3. should be changed so that finished jobs unconditionally transition to NeedsReport the state used to schedule a retry of the report generation in case of report generation failure.

This way when a long job takes a long time generating a report and a following jobs finishes quickly those two don't block the queue and prevent the next job from starting.

[Mark-Simulacrum]

This is somewhat intentional -- since report generation can take a long time (multiple hours, today) we want to avoid building up a queue of jobs needing to generate reports.

My hope is that we can dramatically speed up report generation by moving individual file S3 uploads to happen during the run itself, which should be the majority of the time spent today on report generation.

[Skgland]

Ah, so are e.g. the logs currently buffered so that there is a storage concern? And by uploading logs as they arrive wouldn't that also remove that concern?

[Skgland]

At this point the experiment has been in the generating report state for almost 17 hours, that's a significant amount of time of a fresh check run.

[Mark-Simulacrum]

The logs are currently stored on disk in sqlite (~34GB db as of right now). I'm not too worried about the disk usage, we can allocate larger disks cheaply enough that's not an issue. But it's more that if we let future runs start, then that disk usage isn't capped at whatever ~2 runs take, it's effectively unbounded.

If we upload as we execute, then the report generation should be faster (say, <1 hour) -- note that it needs to read all the logs again most likely, since we produce the .tar.zst bundles, so at least to start we'd still want to keep the local storage into sqlite rather than re-downloading everything from S3. (In the future it could make sense to change that).

[Mark-Simulacrum]

At this point the experiment has been in the generating report state for almost 17 hours, that's a significant amount of time of a fresh check run.

Yeah, that part is I think going to be fixed or improved with #804, we were spending a bunch of time (~1 CPU core) serving requests to fetch next crate, in part because the agent/worker fleet didn't manage to kill a few processes so we hadn't downsized that fleet to the minimal ~3 instances.

[Skgland]

The logs are currently stored on disk in sqlite (~34GB db as of right now). I'm not too worried about the disk usage, we can allocate larger disks cheaply enough that's not an issue. But it's more that if we let future runs start, then that disk usage isn't capped at whatever ~2 runs take, it's effectively unbounded.

Yeah, the unboundedness was what I was referring to with there being a storage concern

If we upload as we execute, then the report generation should be faster (say, <1 hour) -- note that it needs to read all the logs again most likely, since we produce the .tar.zst bundles, so at least to start we'd still want to keep the local storage into sqlite rather than re-downloading everything from S3. (In the future it could make sense to change that).

Understandable, but a bit unfortunate.
Would it make sens to rather than storing the full log to just store links to the full log? Though I suppose not having to download logs individually and getting benefits from compression might be the point of having the bundles in the first place and doing that would defeat the point. A question would be how often these bundles are actually used.

[Mark-Simulacrum]

The goal is precisely that you can quickly grab the full set of logs (at least for regressed crates, but ideally all crates) for local analysis. The tarballs aren't super big and it's much faster than hitting S3 100,000+ times.

[Mark-Simulacrum]

In some sense, if we had a relatively fast UI that served content from the tarballs, I'd rather drop the individual files stored in S3 -- those are (more) expensive to store since they're tiny, and S3's more cost-effective tiers are optimized for big files, not millions of tiny files. But that's its own project :)