[BUG] Background job interval lookup outside error handling stops the host on invalid reload or DB error #281

Summary

Every scheduled background job reads its interval (GetInterval()) outside the error handling of ScheduledBackgroundService. If that read throws — because a reloaded appsettings.json fails validation, or because the observed-image refresh cannot count enabled images in the database — the exception escapes ExecuteAsync, and with the platform's default BackgroundServiceExceptionBehavior.StopHost the whole application stops. A single bad configuration edit or a short database outage can therefore take the application down, and with an invalid file on disk a restarting container then fails startup validation in a loop.

Area

• docs/areas/scan-engine.md (added in [Docs] Describe the current behavior by area #263) — Background jobs, Failure isolation, Open question 1
• docs/areas/hosting.md (added in [Docs] Describe the current behavior by area #263) — Configuration reload, Open question 1
• Security area: None

Current behavior

src/DockerUpdateGuard/Images/ScheduledBackgroundService.cs:127-160 — the interval is read twice outside any try:

if (_logger.IsEnabled(LogLevel.Information))
{
    _logger.BackgroundServiceStarted(backgroundServiceName, GetInterval().TotalMinutes);   // line 133
}
...
while (stoppingToken.IsCancellationRequested == false)
{
    var interval = GetInterval();                                                          // line 145

Only ExecuteStartupAsync (lines 71-90) and ExecuteCoreAsync (lines 97-120) are wrapped in try/catch.

The GetInterval() implementations all read IOptionsMonitor<DockerUpdateGuardOptions>.CurrentValue:

• Images/DockerInstanceDiscoveryBackgroundService.cs:49-52, Images/DockerHubAccountImageDiscoveryBackgroundService.cs:49-52, Images/RuntimeContainerRefreshBackgroundService.cs:50-53, Images/ResourceStatisticsRefreshBackgroundService.cs:49-52, Images/ScanCleanupBackgroundService.cs:144-147
• Images/OwnImageBaseRefreshBackgroundService.cs:51-60 additionally opens a scope and runs a synchronous database query on every call:

var observedImageCount = dbContext.ObservedImages.Count(entity => entity.IsEnabled);
return ObservedImageScanIntervalCalculator.CalculateInterval(_optionsMonitor.CurrentValue.Scanning, observedImageCount);

Options are registered with a validator and ValidateOnStart() (src/DockerUpdateGuard/ServiceCollectionExtensions.cs:53-56). After a reload, IOptionsMonitor.CurrentValue re-creates the options through the options factory, which runs DockerUpdateGuardOptionsValidator and throws OptionsValidationException when validation fails. Nothing is cached on failure, so every subsequent read throws again until the file is fixed — the monitor does not keep the last valid value.

No HostOptions.BackgroundServiceExceptionBehavior is configured (src/DockerUpdateGuard/Program.cs, ServiceCollectionExtensions.cs), so the default StopHost applies.

Reproduction (configuration reload):

1. Start the application with a valid configuration and wait until the jobs are waiting for their next run.
2. Edit the mounted appsettings.json and set DockerUpdateGuard:Scanning:CleanupIntervalMinutes to 0 (allowed range 1–10080).
3. When any job finishes its current wait and runs its next iteration, GetInterval() throws OptionsValidationException at ScheduledBackgroundService.cs:145, the hosted service faults, and the host shuts down. A container restart policy then restarts the process, which fails startup validation with the invalid file.

Reproduction (database): while the observed-image refresh waits for its next run, make the database unreachable (e.g. stop PostgreSQL) for longer than the retrying execution strategy needs to give up. The Count at OwnImageBaseRefreshBackgroundService.cs:56 throws, the exception leaves ExecuteAsync and the host stops — although the scan engine's documented behavior is that a failing run never stops the host (Failure isolation in docs/areas/scan-engine.md).

Expected behavior

• Determining a job's interval never stops the host and never ends the job's loop. Failures are logged and the job keeps running.
• Decision (product manager): on a failing interval lookup the job keeps the last valid configuration / the last known interval, logs the failure, and never stops the host.
• Concretely: each job remembers the last interval it computed successfully. If computing the interval throws (any exception other than cancellation on shutdown), the job logs a warning with the job name and the exception, uses the last known interval for this wait, and tries again before the next wait.
• If no interval has ever been computed successfully (the very first lookup fails), the job uses the interval of the configuration validated at startup (the value that was valid when ValidateOnStart ran), so there is always a defined fallback.
• When the configuration becomes valid again (or the database becomes reachable again), the next lookup succeeds and the new interval takes effect from the next wait, as today.
• The startup log line (BackgroundServiceStarted, event 2000) must not be able to fail the job either.
• Host shutdown during the lookup still ends the job quietly (no error log), as today.

Acceptance criteria

• An exception thrown by GetInterval() — at the start-up log line and inside the wait loop — is caught, logged as a warning with the job name, and the job continues with the last known interval.
• A reloaded configuration that fails validation does not stop the host; jobs keep their last valid interval until a valid configuration is loaded again, after which the new interval is used.
• A database error while the observed-image refresh counts enabled images does not stop the host; the job uses its last known interval.
• If the first interval lookup of a job fails, the job falls back to the interval of the startup-validated configuration.
• Unit tests in src/Tests/DockerUpdateGuard.Tests/ScheduledBackgroundServiceTests.cs (helper Helper/TestScheduledBackgroundService.cs), for example:
  • ScheduledBackgroundServiceIntervalLookupThrowsUsesLastKnownInterval
  • ScheduledBackgroundServiceInitialIntervalLookupThrowsDoesNotFault
  • ScheduledBackgroundServiceIntervalLookupRecoversUsesNewInterval
  • OwnImageBaseRefreshBackgroundServiceObservedImageCountFailsUsesLastKnownInterval
  • a test using a hand-written options monitor whose CurrentValue throws OptionsValidationException proving that the job's ExecuteAsync task does not fault.
    All timing through an injected TimeProvider (see Implementation notes), never the real clock; every assertion has a message.
• Coverage gate passes (≥ 80 % on new/changed lines).
• The area documents docs/areas/scan-engine.md and docs/areas/hosting.md describe the new behavior (fallback to the last known interval, no host stop), and their Known gaps entries for this issue are removed.
• docs/ARCHITECTURE.md (background job engine section) states the behavior on a failing interval lookup.

Implementation notes

• Fix it once in ScheduledBackgroundService (src/DockerUpdateGuard/Images/ScheduledBackgroundService.cs): add a private helper (e.g. GetIntervalSafely) that wraps GetInterval() in try/catch, stores the last successful value in a field, and logs on failure; use it at lines 133 and 145. Do not duplicate the handling in the seven subclasses.
• The startup fallback: the base class has no access to the options. Options: (a) an abstract/virtual GetFallbackInterval() per job, reading IOptions<DockerUpdateGuardOptions>.Value (the startup snapshot, validated by ValidateOnStart), or (b) pass an initial interval into the base constructor. Recommendation: (a), because IOptions<T>.Value is computed once at startup and is never re-validated on reload. For OwnImageBaseRefreshBackgroundService the fallback is OwnImageBaseScanIntervalMinutes (the documented floor).
• Add a new LoggerMessage in Images/ImageHostLoggingExtensions.cs (next free ID in the 20xx background-service range, e.g. 2005, message template with PascalCase parameters such as {BackgroundServiceName} and {FallbackIntervalMinutes}). Do not log configuration values other than the interval.
• Task.Delay(interval, stoppingToken) (line 151) uses the real clock. To make the loop testable without real waiting, inject a TimeProvider (registered as TimeProvider.System in the composition root) and use Task.Delay(interval, timeProvider, token); tests use a hand-written fake TimeProvider (no new package needed). This changes the constructors of all seven jobs — update their registrations and Helper/TestScheduledBackgroundService.cs / Helper/TestScanCleanupBackgroundService.cs.
• Do not set BackgroundServiceExceptionBehavior.Ignore globally as the fix: it would hide the fault but leave the job loop dead (the job would silently never run again).
• Out of scope: other consumers of IOptionsMonitor<DockerUpdateGuardOptions>.CurrentValue (orchestrators, UI services) will also throw while the configuration is invalid; those calls already run inside the jobs' run-level error handling or the UI's error handling and do not stop the host. Mention this in the area document (a run during an invalid configuration fails and is logged).
• Must not change: runs never overlap within one job; a wait already in progress is not shortened or extended; scan cleanup's first full pass still happens only after one interval (ShouldExecuteImmediately() == false).
• No configuration keys, no README configuration reference changes and no migration.

Related

• PR [Docs] Describe the current behavior by area #263 (area documents)
• F-048 (P7) — Scan-resilience defects (F-024 batch abort, F-025 transient finding loss) without regression test #19 (scan-resilience defects) — different failure path (inside runs), but the same "a failure must not stop the batch/host" principle.
• F-020 (P2) — Active provider fixed at startup, but config read live #22 (vulnerability provider fixed at startup while configuration is read live) — another configuration-reload inconsistency.

Generated by Claude Code