diff --git a/README.md b/README.md index 7fa2e2b..e6721a3 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,7 @@ DockerUpdateGuard uses normal ASP.NET Core configuration binding and prefers con - `appsettings.json` - `appsettings.{Environment}.json` +- environment variables (`__` as the section separator, e.g. `DockerUpdateGuard__Scanning__RetryCount`) - command-line arguments - secret stores supported by ASP.NET Core @@ -96,8 +97,11 @@ For example: | `ConnectionStrings:DockerUpdateGuard` | none | Yes* | Named PostgreSQL connection string used when `DockerUpdateGuard:ConnectionString` is not set | | `DockerUpdateGuard:ConnectionString` | none | Yes* | Inline PostgreSQL connection string | | `DockerUpdateGuard:ConnectionStringName` | `DockerUpdateGuard` | No | Name of the `ConnectionStrings` entry to resolve | -| `DockerUpdateGuard:DisplayVersion` | assembly version | No | Version string shown in the UI footer; the Docker image sets this automatically | - +| `DockerUpdateGuard:DisplayVersion` | assembly version | No | Version string shown in the navigation drawer; the Docker image sets this automatically | +| `DockerUpdateGuard:Database:MaxConnectionRetryCount` | `8` | No | Retries of a transiently failing database command during normal operation | +| `DockerUpdateGuard:Database:MaxConnectionRetryDelaySeconds` | `30` | No | Maximum back-off in seconds between those retries | +| `DockerUpdateGuard:Database:MigrationStartupTimeoutSeconds` | `120` | No | How long startup waits for the database to become reachable | +| `DockerUpdateGuard:Database:MigrationRetryDelaySeconds` | `5` | No | Delay in seconds between connection attempts while startup waits for the database | \* At least one of `DockerUpdateGuard:ConnectionString` or `ConnectionStrings:{ConnectionStringName}` must be configured. @@ -109,7 +113,7 @@ For example: | `ApiBaseUrl` | `https://hub.docker.com/` | No | Docker Hub API base address used for registries served by Docker Hub | | `UserName` | none | No | Docker Hub username for authenticated API requests | | `Pat` | none | No | Docker Hub personal access token | -| `RequestTimeoutSeconds` | `30` | No | Timeout for outbound Docker Hub and OCI registry requests | +| `RequestTimeoutSeconds` | `30` | No | Total time per outbound Docker Hub, OCI registry and .NET/nginx release-feed request, including its retries | | `MaxParallelRequests` | `4` | No | Maximum logical request parallelism for registry operations | ### `DockerUpdateGuard:ReleaseMetadata` @@ -149,7 +153,7 @@ versions. Override them to route the lookups through an internal mirror or proxy | `ResourceStatisticsIntervalMinutes` | `5` | Interval for sampling CPU, memory, and network usage | | `VulnerabilityRefreshIntervalMinutes` | `180` | Interval for refreshing vulnerability information | | `CleanupIntervalMinutes` | `720` | Interval for cleaning old scan data | -| `RetryCount` | `2` | Retry count for transient background failures | +| `RetryCount` | `2` | Retries of a single outbound HTTP request after a transient failure (`0` to `10`); applies to Docker Hub, other registries, the release feeds and Portainer, for scheduled and manual work alike. A failed scan run is not retried before its next scheduled run | | `RetainScanRunsDays` | `30` | Retention period for completed scan history | | `MajorUpgradeMinimumAgeDays` | `30` | Waiting period before a new major version line is recommended; the upgrade also stays deferred while the current major version line is still served next to the new line or while its first release is unknown; `0` disables the waiting periods | | `MajorUpgradeMinimumReleaseCount` | `2` | Number of releases a new major version line must have before it is recommended; `0` disables the requirement | @@ -205,9 +209,9 @@ must be configured. | `ServiceName` | `DockerUpdateGuard` | Required when telemetry is enabled | Service name used in telemetry resources | | `OtlpEndpoint` | none | No | Absolute `http` or `https` OTLP endpoint | | `Instance` | none | No | Logical deployment instance name | -| `EnableLogging` | `true` | No | Enables OpenTelemetry logging export | -| `EnableMetrics` | `true` | No | Enables OpenTelemetry metrics export | -| `EnableTracing` | `true` | No | Enables OpenTelemetry tracing export | +| `EnableLogging` | `true` | No | Enables OpenTelemetry logging (exported only when `OtlpEndpoint` is set; console logging is always active) | +| `EnableMetrics` | `true` | No | Enables OpenTelemetry metrics (exported only when `OtlpEndpoint` is set) | +| `EnableTracing` | `true` | No | Enables OpenTelemetry tracing (exported only when `OtlpEndpoint` is set) | If all three telemetry switches are `false`, telemetry is effectively disabled. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 390815e..e920885 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,11 +2,13 @@ This document describes the architecture of DockerUpdateGuard as currently -implemented. It is a technical reference for contributors and AI coding -agents working in this repository; it complements, and does not repeat, -[README.md](../README.md) (product scope, configuration reference), the -[docs/](.) folder (contribution, testing, deployment tiers), and -[AGENTS.md](../AGENTS.md) / +implemented: how the parts fit together and why. It is a technical reference +for contributors and AI coding agents working in this repository. What each +part guarantees — formats, defaults, limits, error behavior — is written once +in the [area documents](areas/README.md) and linked from here instead of +repeated. It also complements [README.md](../README.md) (product scope, +configuration reference), the [docs/](.) folder (contribution, testing, +deployment tiers), and [AGENTS.md](../AGENTS.md) / [.github/copilot-instructions.md](../.github/copilot-instructions.md) and [.github/instructions/csharp.instructions.md](../.github/instructions/csharp.instructions.md) (coding conventions). @@ -15,19 +17,25 @@ agents working in this repository; it complements, and does not repeat, DockerUpdateGuard is a single-process ASP.NET Core web application that: -1. discovers Docker Engine instances and the containers running on them, +1. discovers Docker Engine instances and the containers running on them + ([Docker instances](areas/docker-instances.md)), 2. correlates what is running with registry tag/digest data to detect - available or needed updates, -3. resolves base-image relationships and reuse across observed images, -4. enriches images with vulnerability data (Trivy or Docker Scout), -5. records the results of all of the above as history in PostgreSQL, and -6. serves a Blazor Server UI over that data. + available or needed updates ([Registries](areas/registries.md), + [Image catalog](areas/image-catalog.md), + [Update detection](areas/update-detection.md)), +3. resolves base-image relationships and reuse across observed images + ([Base images](areas/base-images.md)), +4. enriches images with vulnerability data from Trivy or Docker Scout + ([Vulnerabilities](areas/vulnerabilities.md)), +5. records the results of all of the above as history in PostgreSQL + ([Scan engine](areas/scan-engine.md)), and +6. serves a Blazor Server UI over that data ([Web UI](areas/web-ui.md)). There is no separate API tier, message queue, or worker process — scanning, persistence, and UI all run inside one ASP.NET Core host, coordinated in-process via `IHostedService` background jobs and a shared PostgreSQL database. This keeps deployment to a single container plus PostgreSQL (see -§10), at the cost of the background scan engine and the UI sharing the same +§13), at the cost of the background scan engine and the UI sharing the same process and connection pool. ## 2. Solution layout @@ -75,84 +83,70 @@ documented in [AGENTS.md](../AGENTS.md) and not repeated here. registration to `ServiceCollectionExtensions.AddDockerUpdateGuardHost`, the composition root, which in order: -1. Binds and validates `DockerUpdateGuardOptions` (`.ValidateOnStart()` — - the host fails fast on invalid configuration before it accepts traffic). +1. Binds and validates `DockerUpdateGuardOptions` (`.ValidateOnStart()`). 2. Resolves the PostgreSQL connection string (`DockerUpdateGuardConnectionStringResolver`) and registers the EF Core - context via `DockerUpdateGuard.Data`'s `AddDockerUpdateGuardData`, with - `EnableRetryOnFailure` driven by `DatabaseOptions`. + context via `DockerUpdateGuard.Data`'s `AddDockerUpdateGuardData`. 3. Registers OpenTelemetry via `DockerUpdateGuard.Telemetry`'s `AddDockerUpdateGuardTelemetry`. -4. Registers `TransientHttpRetryHandler` as the outbound message handler - for every named `HttpClient` (Docker Hub, generic OCI registry, - .NET/nginx release metadata, Portainer). +4. Registers the typed/named `HttpClient`s (Docker Hub, generic OCI + registry, .NET/nginx release metadata, Portainer) with + `TransientHttpRetryHandler` as their outbound message handler. 5. Registers the integration clients, domain/orchestration services, the UI facade (`IApplicationViewService`), and all 7 background jobs - (`AddHostedService()`) — see §6 and §7. + (`AddHostedService()`) — see §6 to §8. `Program.cs` then adds MudBlazor (`AddMudServices()`) and Razor Components -with **Interactive Server** rendering (`AddInteractiveServerComponents()` / -`AddInteractiveServerRenderMode()`) — Blazor Server only, no WebAssembly or +with **Interactive Server** rendering — Blazor Server only, no WebAssembly or Auto render mode anywhere in the app. After `WebApplication.Build()` and before `RunAsync()`, the host calls `ApplicationInitializationExtensions.InitializeDockerUpdateGuardAsync`, -which blocks startup on: - -1. `DatabaseMigrator.MigrateAsync` — waits for the database to become - reachable (retrying transient connection failures), then, on PostgreSQL, - takes a session-scoped **advisory lock** - (`pg_advisory_lock(5410320540103205)`) before running EF Core migrations - and releases it in a `finally`. This serializes `dotnet ef`-style - migrations across multiple instances of the app starting concurrently - against the same database (e.g. a rolling deployment), so only one - instance applies migrations while the others wait. +which blocks startup on three steps: + +1. `DatabaseMigrator.MigrateAsync` — waits for the database, then applies + EF Core migrations under a PostgreSQL session-scoped advisory lock, so + concurrently starting instances never migrate at the same time. 2. `IInstanceDiscoveryService.SynchronizeConfiguredInstancesAsync` — - reconciles configured `DockerInstanceOptions` against the DB. This is - local, DB-only work and is awaited synchronously. + reconciles the configured Docker instances against the database + (local, DB-only work). 3. `ApplicationTelemetry.RefreshInventoryMetricsAsync` — seeds the - observable gauges (§9) with current counts. - -Docker Hub account discovery (`DockerHubAccountImageDiscoveryService`) is -**deliberately not** part of this blocking sequence, because it makes -sequential per-repository network calls — it instead runs as the first -iteration of `DockerHubAccountImageDiscoveryBackgroundService` immediately -after the host starts accepting traffic, so a slow or unreachable Docker -Hub does not delay application startup. - -The middleware pipeline (non-Development only where noted) is: -`UseExceptionHandler("/error")` → `UseHsts()` → `UseHttpsRedirection()` -(Production only) → `UseStatusCodePagesWithReExecute("/not-found")` → -`UseAntiforgery()` → `MapStaticAssets()` → -`MapRazorComponents().AddInteractiveServerRenderMode()`. There is no -authentication, authorization, health-check, or rate-limiting middleware — -see §11 for the resulting security posture. + inventory gauges (§10). + +Network-bound work (Docker Hub account discovery, scans) is deliberately +not part of this blocking sequence: it runs as the first iteration of the +background jobs once the host accepts traffic (§8), so a slow or +unreachable remote system does not delay startup. + +The exact startup order, retry and timeout values, the middleware pipeline +and error pages, and how configuration reload behaves are specified in +[Hosting and configuration](areas/hosting.md). There is no authentication, +authorization, health-check, or rate-limiting middleware (§11). ## 4. Configuration model Configuration is bound from the `DockerUpdateGuard` and `Telemetry` sections -via standard ASP.NET Core configuration (files, command-line, secret -stores). The full key-by-key reference (defaults, valid ranges, required -combinations) lives in [README.md](../README.md#configuration-reference) and -is not duplicated here. +via standard ASP.NET Core configuration (files, environment variables, +command line). The key-by-key reference lives in +[README.md](../README.md#configuration-reference); which keys are validated, +which apply on reload and which need a restart is specified in +[Hosting and configuration](areas/hosting.md), and the meaning of each +feature's keys in that feature's area. Architecturally relevant points: -- Every options class (`DockerUpdateGuardOptions` and its nested - `Database`/`DockerHub`/`ReleaseMetadata`/`Vulnerabilities`/`Scanning`/ - `DockerInstances[]`/`Portainer` sections, plus `TelemetryOptions`) has a - matching `IValidateOptions` validator wired with `.ValidateOnStart()`, - so misconfiguration is a startup failure, not a runtime surprise. -- `ScanningOptions` and `DockerHubOptions` are read live through - `IOptionsMonitor` by the background jobs and - `TransientHttpRetryHandler`, so interval/retry/parallelism changes in - `appsettings.json` take effect without a restart (a config *reload*, not - a live-edit UI). -- `DockerUpdateGuardConnectionStringResolver` prefers an inline - `DockerUpdateGuard:ConnectionString` over the named `ConnectionStrings` - entry, and is used identically by both the DI registration and the - validator so the two can never disagree about which connection string is - active. +- One `IValidateOptions` validator covers the whole `DockerUpdateGuard` + section and one the `Telemetry` section, both wired with + `.ValidateOnStart()`, so misconfiguration is a startup failure, not a + runtime surprise. +- The background jobs and `TransientHttpRetryHandler` read + `IOptionsMonitor` on every use, so schedule and + retry settings follow a configuration reload. Settings consumed while the + container is composed (`HttpClient` base addresses and timeouts, the + database, telemetry) are fixed until restart. +- `DockerUpdateGuardConnectionStringResolver` is used identically by the DI + registration and the validator, so the two can never disagree about which + connection string is active. ## 5. Data layer (`DockerUpdateGuard.Data`) @@ -163,25 +157,25 @@ auto-discovered via `ApplyConfigurationsFromAssembly`. ### 5.1 Entity model -| Entity | Role | -| --- | --- | -| `RegistryRepository` | Normalized `(Registry, Repository)` pair; parent of all `ImageVersion` rows for that repository. | -| `ImageVersion` | Normalized, deduplicated `(repository, tag, digest)` node — the central catalog entity; carries a vulnerability-assessment rollup. | -| `ImageRelationship` | Edge from a child `ImageVersion` to its base `ImageVersion` (`BaseImage`/`BuildStage`), with `Depth` — the base-image dependency graph. | -| `ObservedImage` | A user/discovery-registered "track this image" target; points at a `CurrentImageVersion`. | -| `DockerInstance` | A configured Docker Engine endpoint; optionally has one `PortainerEndpoint`. | -| `PortainerEndpoint` | Portainer connection details for a `DockerInstance`. | -| `ContainerSnapshot` | A discovered/running container at a point in time, with its resolved `UpdateAssessmentStatus`. | -| `ScanRun` | A persisted scan execution (type/status/trigger), optionally scoped to an `ObservedImage` or `DockerInstance`. | -| `UpdateFinding` | A detected update (subject `ImageVersion` → optional recommended `ImageVersion`), linked to an `ObservedImage` and/or `ContainerSnapshot`. | -| `TagCandidate` | One ranked candidate tag belonging to an `UpdateFinding`. | -| `VulnerabilityFinding` | A CVE/advisory against an `ImageVersion` (severity, fix status, CVSS). | -| `ContainerActionRun` | A requested Portainer action (restart/redeploy/update) against a container, service, or stack. | -| `RuntimeContainerTagSelection` | A manually pinned tag/digest override for a specific runtime container. | -| `DockerInstanceResourceSample` / `RuntimeContainerResourceSample` | Instance-level and per-container CPU/memory/network time series. | +| Entity | Role | Area | +| --- | --- | --- | +| `RegistryRepository` | `(Registry, Repository)` pair; parent of all `ImageVersion` rows for that repository. | [Image catalog](areas/image-catalog.md) | +| `ImageVersion` | Deduplicated `(repository, tag, digest)` node — the central catalog entity; carries a vulnerability-assessment rollup. | [Image catalog](areas/image-catalog.md) | +| `ImageRelationship` | Edge from a child `ImageVersion` to its base `ImageVersion`, with type and `Depth` — the base-image dependency graph. | [Base images](areas/base-images.md) | +| `ObservedImage` | A manually or discovery-registered "track this image" target; points at a `CurrentImageVersion`. | [Image catalog](areas/image-catalog.md) | +| `DockerInstance` | A configured Docker Engine endpoint; optionally has one `PortainerEndpoint`. | [Docker instances](areas/docker-instances.md) | +| `PortainerEndpoint` | Portainer connection details for a `DockerInstance`. | [Container actions](areas/container-actions.md) | +| `ContainerSnapshot` | A discovered container at a point in time, with its `UpdateAssessmentStatus`. | [Docker instances](areas/docker-instances.md) | +| `ScanRun` | A persisted scan execution (type/status/trigger), optionally scoped to an `ObservedImage` or `DockerInstance`. | [Scan engine](areas/scan-engine.md) | +| `UpdateFinding` | A detected update (subject `ImageVersion` → optional recommended `ImageVersion`), linked to an `ObservedImage` and/or `ContainerSnapshot`. | [Update detection](areas/update-detection.md) | +| `TagCandidate` | One ranked candidate tag belonging to an `UpdateFinding`. | [Update detection](areas/update-detection.md) | +| `VulnerabilityFinding` | A CVE/advisory against an `ImageVersion` (severity, fix status, CVSS). | [Vulnerabilities](areas/vulnerabilities.md) | +| `ContainerActionRun` | A record of a Portainer action against a container, service, or stack. | [Container actions](areas/container-actions.md) | +| `RuntimeContainerTagSelection` | A user's tag/digest selection for a specific runtime container. | [Update detection](areas/update-detection.md) | +| `DockerInstanceResourceSample` / `RuntimeContainerResourceSample` | Instance-level and per-container CPU/memory/network time series. | [Docker instances](areas/docker-instances.md) | Every enum used by these entities lives alongside them in the `Entities` -namespace (the dedicated `Enums\` folder is present but unused). +namespace. ### 5.2 Repositories vs. Queries @@ -189,20 +183,19 @@ The `Repositories\` and `Queries\` folders are deliberately split by read/write intent, not by entity: - **`Repositories\` — `IImageCatalogRepository`.** The single write-side - aggregate root for the `RegistryRepository`/`ImageVersion` catalog. Its - `GetOrCreateImageVersionAsync`/`GetOrCreateRegistryRepositoryAsync` - methods enforce catalog uniqueness (including a `DbUpdateException` - race-retry path for concurrent get-or-create calls under load) so - scanners never create duplicate catalog rows. This is the only - repository in the project — no entity outside the catalog needed one. + entry point for the `RegistryRepository`/`ImageVersion` catalog. Its + get-or-create methods enforce catalog uniqueness, including a race-retry + path for concurrent get-or-create calls, so scanners never create + duplicate catalog rows ([Image catalog](areas/image-catalog.md)). This is + the only repository in the project — no entity outside the catalog + needed one. - **`Queries\` — `ILiveImageInventoryQueryService`, `ISharedBaseImageQueryService`.** Read-only, multi-entity, page-shaped - projections (joins/grouping across `ImageVersion`, `ImageRelationship`, - `ObservedImage`, `ContainerSnapshot`) that return plain DTOs instead of - tracked entities, and don't naturally belong to any single entity's - repository. `ILiveImageInventoryQueryService.GetLiveImageVersionIdsAsync` - in particular is the shared definition of "still relevant" used by - `ScanCleanupBackgroundService` (§7) to decide what history to keep. + projections that return plain DTOs instead of tracked entities and don't + naturally belong to any single entity's repository. + `ILiveImageInventoryQueryService` is the shared definition of "still + relevant" that cleanup uses to decide what history to keep + ([Scan engine](areas/scan-engine.md)). ### 5.3 Migrations @@ -216,34 +209,34 @@ DockerUpdateGuardDbContextFactory` provides the design-time `DbContext` for Each external system has one hand-written client behind a narrow interface — there is no dependency on `Docker.DotNet`, a Portainer SDK, or a Trivy -client library. All clients return plain DTOs wrapped in +client library. Clients return plain DTOs wrapped in `ExternalOperationResult` (`Infrastructure\ExternalOperationResult{T}.cs`) -rather than throwing for expected failure modes (`NotConfigured`, -`Unsupported`, `NotFound`, `Failed`, `Unknown`); orchestrators branch on -that status instead of on exceptions. +for expected failure modes, and orchestrators branch on that status instead +of on exceptions. The outcomes and where errors still escape are specified +in [Registries](areas/registries.md#failure-model-of-integrations). -| Client | Talks to | Notes | +| Client | Talks to | Area | | --- | --- | --- | -| `DockerInstanceClient` (`Docker\`) | Docker Engine HTTP API (`v1.41`) | One HTTP transport supporting `http(s)://`, `tcp://`, `unix://` (via `SocketsHttpHandler.ConnectCallback`), and `npipe://` (Windows named pipes) behind a single client, pooled per connection config. | -| `DockerHubClient` (`DockerHub\`) | Docker Hub REST API **and** the OCI Distribution v2 API (`registry-1.docker.io`) | Implements both `IDockerHubClient` (repository/tag metadata) and the generic `IRegistryMetadataClient` (manifest/blob/base-image-label resolution), because Docker Hub is both a metadata API and an OCI registry. | -| `OciRegistryClient` (`Images\`) | Any OCI Distribution v2 registry | The registry-agnostic counterpart to `DockerHubClient`'s registry half, for non-Docker-Hub registries. | -| `PortainerClient` (`Portainer\`) | Portainer REST API | JWT login, then endpoint/container listing and action execution (restart/redeploy/update) behind `IPortainerClient`. | -| `TrivyVulnerabilityProvider` (`Vulnerabilities\`) | Trivy CLI (external process), client mode against a Trivy server | Runs `trivy image --server --format json …` via `IProcessRunner`/`ProcessRunner`, parses stdout JSON. The `trivy` binary ships inside the container image (see §10). | -| `DockerScoutVulnerabilityProvider` (`Vulnerabilities\`) | Docker Scout / Docker Hub HTTP API | Docker Hub login → cached bearer token → paginated vulnerability fetch (capped at 50 pages). | -| `DefaultVulnerabilityProvider` (`Vulnerabilities\`) | — | No-op fallback when vulnerability scanning is disabled/unconfigured (`IVulnerabilityProviderResolver` selects it). | +| `DockerInstanceClient` (`Docker\`) | Docker Engine HTTP API over `http(s)://`, `tcp://`, `unix://` and `npipe://`, one hand-built `HttpClient` pooled per connection config. | [Docker instances](areas/docker-instances.md) | +| `DockerHubClient` (`DockerHub\`) | Docker Hub REST API **and** the OCI Distribution v2 API (`registry-1.docker.io`); implements both `IDockerHubClient` and the generic `IRegistryMetadataClient`, because Docker Hub is both a metadata API and an OCI registry. | [Registries](areas/registries.md) | +| `OciRegistryClient` (`Images\`) | Any other OCI Distribution v2 registry. | [Registries](areas/registries.md) | +| `PortainerClient` (`Portainer\`) | Portainer REST API (API key or JWT login), endpoint and container listing, container actions. | [Container actions](areas/container-actions.md) | +| `TrivyVulnerabilityProvider` (`Vulnerabilities\`) | Trivy CLI (external process via `IProcessRunner`), client mode against a Trivy server. | [Vulnerabilities](areas/vulnerabilities.md) | +| `DockerScoutVulnerabilityProvider` (`Vulnerabilities\`) | Docker Scout / Docker Hub HTTP API. | [Vulnerabilities](areas/vulnerabilities.md) | +| `DefaultVulnerabilityProvider` (`Vulnerabilities\`) | — (no-op fallback selected by `IVulnerabilityProviderResolver` when scanning is disabled). | [Vulnerabilities](areas/vulnerabilities.md) | `RegistryMetadataService` is the dispatcher in front of `IRegistryMetadataClient`: it holds every registered client and picks the first whose `CanHandle(registry)` matches, falling back to -`NullRegistryMetadataClient` (an `Unsupported` sentinel) for registries -nothing can handle — adding a new registry type means adding one more -`IRegistryMetadataClient` implementation, not touching the dispatcher. +`NullRegistryMetadataClient` (an `Unsupported` sentinel) — adding a new +registry type means adding one more `IRegistryMetadataClient` +implementation, not touching the dispatcher. -`TransientHttpRetryHandler` (`Infrastructure\`) is attached to every -outbound `HttpClient` (Docker Hub, OCI registry, release-metadata feeds, -Portainer) and retries transient failures (408/429/5xx, -`HttpRequestException`, `TimeoutException`) honoring `Retry-After`, -otherwise exponential backoff with jitter, up to `ScanningOptions.RetryCount`. +`TransientHttpRetryHandler` (`Infrastructure\`) is attached to the +typed/named `HttpClient`s registered in the composition root (§3). The +Docker Engine client and Docker Scout create their `HttpClient`s without +it. The retry policy is specified in +[Registries](areas/registries.md#retry-of-transient-http-failures). ## 7. Domain / orchestration layer and the scan engine @@ -266,7 +259,7 @@ flowchart TD ISO[ImageScanOrchestrator] RCSO[RuntimeContainerScanOrchestrator] VES[VulnerabilityEnrichmentService] - BIR["IBaseImageResolver
(DockerHub/Registry)"] + BIR["IBaseImageResolver
(registry labels)"] end subgraph Persistence["DockerUpdateGuard.Data"] @@ -287,35 +280,24 @@ flowchart TD ``` - **`IUpdateDetectionService` (`UpdateDetectionService`)** is a pure - evaluator (`Evaluate(ImageReference, tags) -> UpdateEvaluationResult`, - no I/O), so update logic can be unit-tested without mocking HTTP or the - database. -- **`VersionTagResolutionHelper`** is the tag-comparison engine: parses - `major.minor.patch[-suffix]` and `YYYY-suffix` (year-prefixed) tags, - groups tags into "variant families" (e.g. `-alpine`, `-jammy`) so an - Alpine tag is never recommended as an update for a Debian one, resolves - floating alias tags (`latest`) back to the concrete semver tag sharing - their digest, and ranks pre-release identifiers below GA releases. -- **Major-version upgrades are deliberately conservative.** A new major - line is only surfaced as an actionable update once it has at least - `Scanning:MajorUpgradeMinimumReleaseCount` releases, its first (`x.0.0`) - release is older than `Scanning:MajorUpgradeMinimumAgeDays`, and the - current major line has gone dormant. Until then it is shown only as - descriptive text on an otherwise `UpToDate` result — this avoids - recommending a major upgrade the day it ships. Both thresholds accept - `0` to disable the corresponding wait. -- **`IDerivedBaseRuntimeDetector`** infers a locally running .NET/nginx - runtime version from Docker inspect environment variables and image - history alone (no registry call), which `IDotNetReleaseMetadataService`/ - `INginxReleaseMetadataService` then compare against upstream release - feeds — this covers images that don't expose a useful base-image label, - at the cost of only recognizing the two runtimes it has explicit - detection rules for. + evaluator (no network or database I/O) over an image reference and its + tags, and **`VersionTagResolutionHelper`** is the tag-comparison engine + behind it, so update logic — including the deliberately conservative + major-upgrade gating — can be unit-tested without mocking HTTP or the + database. The rules themselves are in + [Update detection](areas/update-detection.md). +- **`IBaseImageResolver`** (registered as `RegistryBaseImageResolver`) + resolves base images from registry image labels; **`IDerivedBaseRuntimeDetector`** + infers a .NET/nginx runtime from environment variables (and, for runtime + containers, image history), which `IDotNetReleaseMetadataService`/ + `INginxReleaseMetadataService` compare against upstream release feeds. + This covers images without a useful base-image label, at the cost of only + recognizing the two runtimes it has explicit rules for + ([Base images](areas/base-images.md)). - **`ImageScanOrchestrator`** guards against overlapping scans of the same - observed image with a static per-image `SemaphoreSlim` registry - (`ObservedImageScanLockEntry`), not a database lock — this is a - same-process concurrency guard, not a multi-instance one (see §11 for - the deployment assumption this relies on). + observed image with a static per-image `SemaphoreSlim` registry, not a + database lock — a same-process concurrency guard, not a multi-instance + one (§11, [Scan engine](areas/scan-engine.md)). - Orchestrators write directly to `DbContext` sets (`UpdateFindings`, `ImageRelationships`, `ScanRuns`) for their own aggregates, and go through `IImageCatalogRepository` only for the shared, dedup-sensitive @@ -329,52 +311,26 @@ All seven scheduled jobs derive from `ScheduledBackgroundService` `ExecuteCoreAsync`, optionally `ExecuteStartupAsync` (runs once before the loop) and override `ShouldExecuteImmediately()` (default `true`). The base class re-evaluates `GetInterval()` every iteration (so config-reload -changes apply live), creates a DI scope per run, and catches/logs -exceptions from `ExecuteCoreAsync` so one failing run never crashes the -host. +changes apply live), creates a DI scope per run, and catches and logs +exceptions from `ExecuteCoreAsync`, so a failing run does not end the job. -| Job | Does | Interval source | -| --- | --- | --- | -| `DockerInstanceDiscoveryBackgroundService` | Reconciles configured `DockerInstanceOptions` against the DB. | `Scanning:DiscoveryIntervalMinutes` | -| `DockerHubAccountImageDiscoveryBackgroundService` | Discovers repositories/images under the configured Docker Hub account. Runs immediately at startup (§3). | `Scanning:DockerHubAccountDiscoveryIntervalMinutes` | -| `OwnImageBaseRefreshBackgroundService` | Scans all enabled observed images for base-image/tag updates (`ImageScanOrchestrator.ScanAllAsync`). | **Dynamic** — see below | -| `RuntimeContainerRefreshBackgroundService` | Scans running containers for update opportunities (`RuntimeContainerScanOrchestrator.ScanAllAsync`). | `Scanning:RuntimeImageUpdateScanIntervalMinutes` | -| `ResourceStatisticsRefreshBackgroundService` | Samples Docker instance/container CPU/memory/network usage. | `Scanning:ResourceStatisticsIntervalMinutes` | -| `VulnerabilityRefreshBackgroundService` | Refreshes CVE findings via the resolved `IVulnerabilityProvider`. | `Scanning:VulnerabilityRefreshIntervalMinutes` | -| `ScanCleanupBackgroundService` | Repairs stale `Running` scan rows on startup, then purges history past retention (see below). | `Scanning:CleanupIntervalMinutes` | - -**Dynamic own-image scan interval.** Unlike the other jobs, -`OwnImageBaseRefreshBackgroundService.GetInterval()` computes its interval -per iteration via `ObservedImageScanIntervalCalculator`: -`max(OwnImageBaseScanIntervalMinutes, ⌈(DockerHubRequestLimitWindowHours × -60 × enabledObservedImageCount) / (DockerHubRequestLimitPerWindow − -DockerHubReservedManualRequestsPerWindow)⌉)`. As the number of tracked -images grows, the interval stretches so scheduled Docker Hub calls stay -within the configured request budget for the window while always reserving -`DockerHubReservedManualRequestsPerWindow` capacity for manual/ad-hoc -scans. This is one of three independent Docker Hub rate-limiting -mechanisms: - -1. **Schedule-level** — the dynamic interval above. -2. **Concurrency-level** — `DockerHubClient` bounds concurrent outbound - calls with a `SemaphoreSlim` sized to `DockerHub:MaxParallelRequests`. -3. **Transport-level** — `TransientHttpRetryHandler` (§6) treats HTTP 429 - as transient and retries with backoff/`Retry-After`. - -There is no separate token-bucket or sliding-window counter; the three -mechanisms above compose to keep Docker Hub traffic within budget. - -**Cleanup and crash recovery.** `ScanCleanupBackgroundService` does not run -immediately; its `ExecuteStartupAsync` first repairs any `ScanRun` stuck in -`Running` status (from a crash or restart) older than -`max(2 × the relevant scan interval, 24h)`, marking it `Failed`. Its -recurring pass then purges data older than `Scanning:RetainScanRunsDays`: -resolved/stale `TagCandidates`, resolved `UpdateFindings`, dead -`VulnerabilityFindings` (immediately if the image is no longer "live" per -`ILiveImageInventoryQueryService`, else age-based), orphaned -`ContainerSnapshots`, and old resource samples — while always keeping the -most recent 20 completed `ScanRun`s for the Scan History UI even past -retention. It finishes by refreshing the telemetry gauges (§9). +| Job | Does | +| --- | --- | +| `DockerInstanceDiscoveryBackgroundService` | Reconciles configured Docker instances against the DB. | +| `DockerHubAccountImageDiscoveryBackgroundService` | Discovers repositories/images under the configured Docker Hub account. | +| `OwnImageBaseRefreshBackgroundService` | Scans all enabled observed images (`ImageScanOrchestrator.ScanAllAsync`) on a dynamic interval. | +| `RuntimeContainerRefreshBackgroundService` | Scans running containers (`RuntimeContainerScanOrchestrator.ScanAllAsync`). | +| `ResourceStatisticsRefreshBackgroundService` | Samples Docker instance/container CPU/memory/network usage. | +| `VulnerabilityRefreshBackgroundService` | Refreshes CVE findings via the resolved `IVulnerabilityProvider`. | +| `ScanCleanupBackgroundService` | Repairs stale `Running` scan rows on startup, then purges history past retention. | + +Intervals and their configuration keys, the dynamic own-image interval that +stretches with the number of observed images to stay within the Docker Hub +request budget, overlap rules, scan-run statuses, crash recovery and +retention are specified in [Scan engine](areas/scan-engine.md). Docker Hub +traffic is shaped by that schedule, by a per-client concurrency limit and +by the transport retry (§6); none of them counts requests. Their exact +behavior is in [Registries](areas/registries.md). ## 9. UI layer @@ -382,92 +338,68 @@ Blazor Server (**Interactive Server** render mode, applied at the `App` root — no WebAssembly, no Auto mode) with **MudBlazor** as the component library, plus a hand-rolled CSS design-system layer (`wwwroot\app.css`, light/dark via a `.dug-dark` class) on top of it. +Pages, routes and what each page shows or triggers are specified in +[Web UI](areas/web-ui.md). -- **Pages** (`Components\Pages`, one `@page` route each): Dashboard (`/`), - My Images / My Image Detail, Observed Images / Observed Image Detail, - Runtime Containers / Runtime Container Detail, Docker Instances / Docker - Instance Detail, Shared Base Images (`/base-images`), Vulnerabilities, - Scan History, plus `Error`/`NotFound`. +- **Pages** (`Components\Pages`) have one `@page` route each. - **`IApplicationViewService` (`UI\ApplicationViewService`)** is the single facade every page depends on. It maps `DockerUpdateGuard.Data` entities - into a set of `*ViewData` DTOs (`UI\`) — pages never query + into `*ViewData` DTOs (`UI\`) — pages never query `DockerUpdateGuardDbContext` or the repositories/query services directly. This keeps EF Core change-tracking and entity shapes out of the component tree, at the cost of `ApplicationViewService` being a fairly wide surface (one method per page/section). - **`Components\Shared`** holds cross-page building blocks: loading - skeletons (`PageHeaderSkeleton`, `TableSkeleton`, `MetricGridSkeleton`) - and the vulnerability widgets (`VulnerabilityAssessmentCard`, - `VulnerabilityFindingsTable`, `VulnerabilitySeverityChips`, - `VulnerabilityRescanControl`) shared between the Vulnerabilities page and - image/container detail pages. -- **`DashboardRefreshState`** is a small injected scoped pub/sub class - background jobs and manual-scan actions use to signal the Dashboard to - refresh without a full page reload. + skeletons and the vulnerability widgets shared between the + Vulnerabilities page and image/container detail pages. +- **`DashboardRefreshState`** is a small singleton pub/sub class that + manual-scan actions in the UI use to signal open Dashboards to refresh + without a full page reload. ## 10. Telemetry `DockerUpdateGuard.Telemetry.AddDockerUpdateGuardTelemetry` wires -OpenTelemetry logging, tracing, and metrics — each independently -toggleable (`Telemetry:EnableLogging/EnableMetrics/EnableTracing`, default -on) — exported over a single OTLP exporter when `Telemetry:OtlpEndpoint` is -a valid absolute HTTP(S) URI (registration is skipped entirely if all three -signals are disabled; there is no console/Prometheus exporter fallback). - -- **Tracing**: a custom `ActivitySource` (`DockerUpdateGuardTelemetry. - ActivitySource`) plus ASP.NET Core and HttpClient instrumentation. - `TelemetryActivityNames` defines the custom activity names in use - (`scan.run`, `dockerhub.request`, `dockerengine.request`, `cve.request`, - `portainer.request`, `portainer.action`, `persistence.operation`). -- **Metrics**: a custom `Meter` plus ASP.NET Core, HttpClient, and .NET - runtime instrumentation. `ApplicationTelemetry` (host project) owns the - application-specific instruments: counters/histogram for scan - runs/failures/duration, and observable gauges for inventory counts - (`ObservedImages`, `RuntimeContainers`, `DeduplicatedBaseImages`, - `ActiveUpdateFindings`, `ActiveCveFindings`, `NeedsReviewFindings`) — - refreshed at startup and after every cleanup pass (§8). -- **Logging**: structured, via `ILogger` and source-generated - `[LoggerMessage]` partial classes — every integration client and - background job has a matching `*Logging.cs` partial class following this - pattern, rather than ad-hoc `_logger.LogInformation(...)` calls. +OpenTelemetry logging, tracing, and metrics, exported over a single OTLP +exporter. A custom `ActivitySource` and `Meter` live in the telemetry +project (`DockerUpdateGuardTelemetry`, `TelemetryActivityNames`); +`ApplicationTelemetry` in the host project owns the application-specific +instruments. Logging is structured, via `ILogger` and source-generated +`[LoggerMessage]` partial classes — every integration client and +background job has a matching `*Logging.cs` partial class rather than +ad-hoc `_logger.LogInformation(...)` calls. Switches, export conditions, +the emitted span and instrument names, and when the inventory gauges +refresh are specified in [Telemetry](areas/telemetry.md). ## 11. Security posture and operating assumptions -These are observed characteristics of the current implementation, not -aspirational guarantees: +These are observed characteristics of the current implementation; the +details are in [Hosting and configuration](areas/hosting.md) and the areas +it links: -- **No authentication or authorization layer.** There is no - `AddAuthentication`/`AddAuthorization`, no login page, and no API key - check anywhere in the host. The application is meant to be deployed - behind network-level access control (private network, VPN, or a reverse - proxy that terminates auth) rather than exposed directly — this is a - deployment responsibility, not something the app enforces itself. See +- **No authentication or authorization layer.** The application is meant + to be deployed behind network-level access control (private network, + VPN, or a reverse proxy that terminates auth) — a deployment + responsibility, not something the app enforces. See [SECURITY.md](../SECURITY.md) for the vulnerability-reporting process. - **No health-check or rate-limiting endpoints** are registered. -- **Single-instance assumption for the scan engine.** The PostgreSQL - advisory lock (§3) serializes *migrations* across concurrently starting - instances, but the background jobs and `ImageScanOrchestrator`'s - in-memory scan lock (§7) are not distributed — running more than one - instance of the app against the same database will duplicate scan work - (harmless but wasteful; catalog writes stay correct via - `IImageCatalogRepository`'s get-or-create race-retry) rather than - coordinate it. -- **Container runs as a non-root, unprivileged user** (UID 64000, GID 0 — - the root group, not root user) so a group-owned, root:root Docker socket - bind-mount stays readable (e.g. on Synology DSM) without running the - process as root; see the comment block in +- **Single-instance assumption for the scan engine.** The advisory lock + (§3) serializes *migrations* across concurrently starting instances, but + the background jobs and `ImageScanOrchestrator`'s in-memory scan lock + (§7) are not distributed — running more than one instance against the + same database duplicates scan work rather than coordinating it; catalog + writes stay correct via `IImageCatalogRepository`'s race-retry. +- **Container runs as a non-root user** (UID 64000, GID 0) so a + group-owned, root:root Docker socket bind-mount stays readable without + running the process as root; see [src/DockerUpdateGuard/Dockerfile](../src/DockerUpdateGuard/Dockerfile). - Mounting the host's Docker socket is nonetheless a high-trust operation - (equivalent to root on the host) and is inherent to the feature, not a - bug. + Mounting the host's Docker socket is nonetheless root-equivalent on the + host and inherent to the feature. - **Secrets in configuration.** Docker Hub PAT, Portainer credentials/API - token, and the database password are plain configuration values (no - secret-store integration beyond what ASP.NET Core's configuration - providers already support). `SkipCertificateValidation` (Docker - instances) and `AllowInsecureHttp` (Portainer) exist for - trusted/self-signed/local setups and are logged as warnings at runtime - when active — operators are responsible for not enabling them over - untrusted networks. + token, and the database password are plain configuration values. + `SkipCertificateValidation` (Docker instances) and `AllowInsecureHttp` + (Portainer) exist for trusted/self-signed/local setups; how each is + validated and reported is in [Docker instances](areas/docker-instances.md) + and [Container actions](areas/container-actions.md). ## 12. Testing architecture @@ -497,14 +429,17 @@ the two production projects that matter for this split: `v*.*.*` tag — builds, tests, then builds and pushes the multi-stage Docker image (tagged with the version and `latest`) to Docker Hub (`networlddev/dockerupdateguard`) and creates a GitHub Release with - auto-generated notes. The image build records the exact - `mcr.microsoft.com/dotnet/runtime` base digest as an OCI label. + auto-generated notes. The workflow passes the digest of + `mcr.microsoft.com/dotnet/runtime:10.0-alpine` into the image's + `org.opencontainers.image.base.digest` label, while the runtime stage + builds on `mcr.microsoft.com/dotnet/aspnet:10.0-alpine` + ([#37](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/37)). - **Container image** (`src/DockerUpdateGuard/Dockerfile`): multi-stage — SDK build stage, - a `trivy` CLI stage copied in from the official Trivy image (bundling - the CLI so `Vulnerabilities:Provider=Trivy` works out of the box against - a separately-run Trivy *server*), and an Alpine ASP.NET runtime stage - running as the non-root user described in §11. Listens on `:8080`. + a `trivy` CLI stage copied in from the official Trivy image (so the + Trivy provider works out of the box against a separately-run Trivy + *server*), and an Alpine ASP.NET runtime stage running as the non-root + user described in §11; see [Hosting and configuration](areas/hosting.md). - **Deployment tiers** (`docs/docker-compose.{minimal,trivy,full}.yml`, walked through in [INSTALL.md](../INSTALL.md)): minimal (app + PostgreSQL), + Trivy server, or + a full Grafana OTel-LGTM stack @@ -520,9 +455,9 @@ Summarized here for quick reference; details are in the sections above. | Decision | Rationale evidenced in the code | | --- | --- | | Blazor Server (Interactive Server), not WebAssembly | Keeps all data access server-side against `DockerUpdateGuardDbContext`; no separate public API surface is needed since there is no WASM client to serve one to. | -| Hand-written HTTP clients instead of `Docker.DotNet`/a Portainer SDK/a Trivy client library | Each client only needs a small slice of its target API (engine stats/inspect, Hub tags, Portainer actions, Trivy CLI JSON) — see §6 for exactly what each one covers. | +| Hand-written HTTP clients instead of `Docker.DotNet`/a Portainer SDK/a Trivy client library | Each client only needs a small slice of its target API — see §6 for what each one covers. | | Trivy runs as an external CLI process (`IProcessRunner`), not an HTTP client to a Trivy server API | The image is scanned locally by the CLI in client mode; only the vulnerability database lookup goes to the configured Trivy server. | -| Update detection (`IUpdateDetectionService`) and tag comparison (`VersionTagResolutionHelper`) are pure, I/O-free | Lets the update-recommendation logic — including the deliberately conservative major-version-upgrade gating (§7) — be unit-tested exhaustively without mocking HTTP or the database. | +| Update detection (`IUpdateDetectionService`) and tag comparison (`VersionTagResolutionHelper`) are pure, I/O-free | Lets the update-recommendation logic — including the deliberately conservative major-upgrade gating — be unit-tested exhaustively without mocking HTTP or the database. | | One shared `IImageCatalogRepository` for the `RegistryRepository`/`ImageVersion` catalog, but no repository for any other entity | Only the catalog has a get-or-create-with-dedup concern that benefits from a single enforced entry point (§5.2); other entities are written directly by the orchestrator that owns them. | | PostgreSQL advisory lock around migrations, but no distributed lock around scanning | Migrations are unsafe to run concurrently and are cheap to serialize once at startup; duplicate concurrent scans are wasteful but not unsafe (§11), so they were not made distributed. | | No authentication/authorization in the app itself | The app assumes a trusted network perimeter (reverse proxy, VPN, private network) rather than implementing its own auth layer (§11). | diff --git a/docs/areas/README.md b/docs/areas/README.md index 7151574..61e31ab 100644 --- a/docs/areas/README.md +++ b/docs/areas/README.md @@ -27,4 +27,15 @@ operator or another component can rely on — formats, limits, error behavior, g | Area | Scope | Not here | | ---- | ----- | -------- | +| [Hosting and configuration](hosting.md) | Loading and validating configuration, startup sequence and database migrations, HTTP pipeline and error pages, container image and runtime user, security posture | Meaning of feature-specific keys (their areas); full key reference (`README.md`); build, CI and release (`CONTRIBUTING.md`) | +| [Docker instances](docker-instances.md) | Docker Engine connectivity and transports, certificate relaxation, discovery of running containers and their runtime status, resource statistics | Update evaluation of containers (Update detection); Portainer actions (Container actions); schedules (Scan engine) | +| [Registries](registries.md) | Read access to Docker Hub and OCI registries, registry dispatch, Docker Hub account discovery and request budget, retry of transient HTTP failures, failure model of all integrations | When scans run (Scan engine); Docker Scout (Vulnerabilities); base-image resolution (Base images) | +| [Image catalog](image-catalog.md) | Image reference parsing and normalization, the deduplicated registry/repository/tag/digest catalog, observed images and how they are registered | Base images (Base images); tag comparison (Update detection); CVEs (Vulnerabilities) | +| [Base images](base-images.md) | Base-image resolution, the base-image relationship graph, shared base images, derived .NET/nginx runtime detection against release feeds | Tag ranking (Update detection); registry access (Registries) | +| [Update detection](update-detection.md) | Tag formats and variant families, alias resolution, pre-release ranking, major-upgrade gating, update assessment, findings and tag candidates, manual tag selection | Fetching tags (Registries); retention (Scan engine) | +| [Vulnerabilities](vulnerabilities.md) | Provider choice (Trivy, Docker Scout, none), assessment status, severities, fix status and CVSS, manual rescan, configuration hints | Schedule and cleanup (Scan engine); display (Web UI) | +| [Container actions](container-actions.md) | Portainer endpoints and authentication, insecure HTTP opt-in, restart/redeploy/update actions and their recorded runs | Container discovery (Docker instances) | +| [Scan engine](scan-engine.md) | Background jobs and their intervals, dynamic own-image interval, overlap rules, scan runs, crash recovery, retention and cleanup, single-instance assumption | What a scan evaluates (the feature areas) | +| [Web UI](web-ui.md) | Pages and routes, what each page shows and triggers, dashboard refresh, error and not-found pages, theme | How the shown values are computed (the feature areas) | +| [Telemetry](telemetry.md) | Logs, traces and metrics, OTLP export, activity and instrument names, inventory gauges, what never reaches telemetry | Deployment of the observability stack (`INSTALL.md`) | diff --git a/docs/areas/base-images.md b/docs/areas/base-images.md new file mode 100644 index 0000000..640593e --- /dev/null +++ b/docs/areas/base-images.md @@ -0,0 +1,145 @@ +# Base images + +## Scope + +Which images an observed image is built on, and what follows from that: the observed-image scan that resolves the +base-image chain from image labels, the base-image relationship graph it stores, the shared base images view that shows +which observed images and running containers depend on the same base, and the derived runtime detection that recognizes +a .NET or nginx runtime inside an image and compares it with the upstream release feeds. Users rely on it to see where +a base-image update or vulnerability reaches; vulnerability refreshes and the dashboard rely on the stored graph. + +## Behavior + +### Observed-image scan + +- An observed-image scan works on the observed image's current image version ([Image catalog](image-catalog.md)) in + three steps, each reading registry data as described in [Registries](registries.md): + 1. **Tag refresh** — a tag lookup for the current version. On success its publication time and tag metadata are + stored on the version. Otherwise the scan becomes *failed* (outcome *failed*) or *partial* (any other outcome). + 2. **Base-image resolution** — described below. + 3. **Derived runtime check** — described below. +- The scan ends *succeeded* when no step reported a problem, *partial* or *failed* as the steps decide (*failed* is + never lowered to *partial*). An unexpected error in any step makes the scan *failed*. The messages of all steps are + stored on the scan run, each distinct message once. When scans run, overlap rules and scan-run bookkeeping: + [Scan engine](scan-engine.md). + +### Resolving the chain + +- Base images are resolved only for observed images. Running containers are not resolved themselves; a container + joins an observed image's chain only when it runs the exact image version — tag and digest — of that observed image. + A container on another digest of the same tag has no base images. +- The source is the image configuration's labels: `org.opencontainers.image.base.name` names the base image (parsed as + an image reference, see [Image catalog](image-catalog.md)); `org.opencontainers.image.base.digest` supplies its + digest (lowercased) when the name carries none. The same rule applies to Docker Hub and to every other registry. +- The chain is followed by reading each base image's own configuration and labels. Depth 1 is the direct base; the + chain ends at depth 5, at an image without the label, or at a base whose configuration cannot be read (the chain + found so far is kept). +- An image without the label has an empty chain; that is a successful resolution. +- When the observed image's own configuration cannot be read, resolution is unavailable: the stored chain is left + unchanged and the scan becomes *failed* (outcome *failed*) or *partial* (any other outcome). +- Only base-image relationships are produced. The relationship type for build stages exists in storage but nothing + records it. + +### Storing the chain + +- A successful resolution replaces the stored base-image relationships of the current version: the previous ones are + removed and the new chain (possibly empty) is recorded. Relationships belong to the image version, so those of an + earlier current version remain when the observed image moves to a new version. +- For each base image a tag lookup refreshes the exact base. Its stored tag is: + - the label's tag when it is a specific version tag (semantic or year-prefixed, see + [Update detection](update-detection.md)) or when no digest is known; + - otherwise (a floating tag such as `latest` with a digest) the specific version tag of the same repository that + points at the same digest, searched among at most 150 tags — restricted to the floating tag's version line unless + it is `latest`, and to tags published since the floating tag itself; when none is found, the label's tag. +- The base image's catalog entry is the stored tag plus the digest. When the tag lookup succeeded, its publication + time and tag metadata are stored on that entry; when it did not, the relationship is still recorded and the scan + becomes *partial* (unless already *failed*) with the lookup's message. +- A relationship records the child (the observed image's current version), the base, the depth, the source reference + (the image whose label named this base: the observed image for depth 1, the previous base for deeper levels) and the + scan run that recorded it (cleared when that run is deleted). +- A relationship is unique per child, base, depth and type; depth is greater than zero and child and base differ. + +### Shared base images + +- `/base-images` lists every base image recorded for the current version of any observed image (enabled or not), at + every depth. An entry used by a single observed image is listed too. +- Entries are grouped case-insensitively: a base with a digest by `registry/repository@digest` (tags that resolve to + the same digest merge); a base without a digest by `registry/repository:tag` together with its source reference. +- Each entry shows the number of distinct observed images using it, the references of the images that name it, the + running containers whose latest snapshot is exactly one of those image versions, and a vulnerability summary of the + base ([Vulnerabilities](vulnerabilities.md)). +- Entries are ordered by number of running containers, then number of observed images (both descending), then by + reference. + +### Derived runtime detection + +- Recognizes the .NET or nginx runtime an image is built on, also without base-image labels. One runtime per image. +- Inputs: for observed images the environment variables of the image configuration from the registry; for running + containers the environment of the local image and its build history (each entry's command and comment) from the + Docker Engine ([Docker instances](docker-instances.md)). Registry images have no build history, so the + history-based signals apply to running containers only. +- Signals, strongest first; on equal strength environment variables win over history, and earlier history entries + over later ones: + + | Strength | Signal | + | -------- | ------ | + | 0 | Environment variable `DOTNET_VERSION` | + | 1 | Environment variable `NGINX_VERSION`; `DOTNET_VERSION` in history | + | 2 | Environment variable `ASPNET_VERSION`; `NGINX_VERSION` in history | + | 3 | `ASPNET_VERSION` in history | + | 4 | A history reference to `mcr.microsoft.com/dotnet/aspnet`, `runtime`, `runtime-deps` or `sdk` with tag `x.y.z` | + +- Variable names are matched case-insensitively; in history the name must be followed by `=`, `:` or whitespace and a + version. The version is the first `major.minor.patch` in the value (anything after it is ignored); a value without + three numeric parts is no signal. The channel is `major.minor`. + +### Release feeds and runtime findings + +- **.NET:** `releases-index.json` under `DockerUpdateGuard:ReleaseMetadata:DotNetBaseUrl` (default + `https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/`). Per channel (`channel-version`) the latest + runtime (`latest-runtime`) and whether that release is a security release (`security`). +- **nginx:** the `download/` page under `DockerUpdateGuard:ReleaseMetadata:NginxBaseUrl` (default + `https://nginx.org/`). Every `nginx-x.y.z.tar.gz` link counts; a channel's latest is its highest version. +- Both URLs are fixed at startup and must be absolute `http`/`https` URLs ([Hosting](hosting.md)). A feed is loaded at + most once per scan and not kept between scans. A channel missing from the feed gives no finding. +- A runtime is outdated when the feed's latest version of the same channel is higher than the detected one. Only + patch updates within the channel are reported; a newer channel is never suggested. +- An outdated runtime yields an update finding of type *derived base runtime update*: + - on an observed image: summary "Own image uses an outdated .NET base runtime" (discovered image) or "Observed image + uses an outdated .NET base runtime" (manual image), likewise for NGINX. The details name the detected and the + latest version, add for .NET that the latest release includes security fixes when it does, and advise to rebuild + and republish (own image) or that the upstream publisher must rebuild (manual image); + - on a running container: the same check, made only when the container's update evaluation completed + ([Update detection](update-detection.md)); the summary says "Own image …" when the container runs a discovered + image's repository, otherwise "A linked image uses an outdated … base runtime". +- On an observed image the previous findings are replaced: with no runtime detected they are removed; with a runtime + detected and the feed available they are removed and a new one is added if still outdated; when the configuration + or the feed cannot be read they stay as they are. + +## Limits and errors + +- Chain depth is capped at 5; base-tag resolution searches at most 150 tags. +- Registry outcomes other than *succeeded* never abort the scan; they set *partial* or *failed* and add the registry's + message (status code and response body, see [Registries](registries.md)). +- Feed failures (non-success status, missing `releases-index` array, unreadable content) only suppress the runtime + finding; they do not change the scan status. Unreadable content is logged as a warning. + +### Known gaps + +- The relationship type for build stages is defined in storage but never produced. Tracked in + [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). +- The shared base images view also lists bases used by a single observed image and counts disabled observed images. + Tracked in [#286](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/286). + +## Related decisions + +None yet. + +## Not here + +- Reading configurations, tags and labels from registries, retry and timeouts: [Registries](registries.md). +- Parsing references and the catalog entries: [Image catalog](image-catalog.md). +- Tag formats, version lines and update recommendations: [Update detection](update-detection.md). +- Vulnerabilities of base images: [Vulnerabilities](vulnerabilities.md). +- Schedules, overlap rules and scan-run retention: [Scan engine](scan-engine.md). +- Page layout of `/base-images` and the detail pages: [Web UI](web-ui.md). diff --git a/docs/areas/container-actions.md b/docs/areas/container-actions.md new file mode 100644 index 0000000..2b2118b --- /dev/null +++ b/docs/areas/container-actions.md @@ -0,0 +1,116 @@ +# Container actions + +## Scope + +The optional Portainer integration of a Docker instance: how a Portainer server is configured and mapped to a Docker +instance, how DockerUpdateGuard authenticates against it, which container actions it can execute through it, and the +stored records reserved for action runs. Operators rely on it to configure Portainer access without weakening +transport security; the UI is the intended caller of the actions. + +## Behavior + +### Configuration and mapping + +- Portainer is configured per Docker instance under `DockerUpdateGuard:DockerInstances::Portainer` with the keys + `Enabled` (default `false`), `BaseUrl`, `Username`, `Password`, `ApiToken`, `EndpointId`, `AllowInsecureHttp` + (default `false`) and `RequestTimeoutSeconds` (default `15`). +- A Docker instance has at most one Portainer endpoint. It is synchronized together with the instance + ([Docker instances](docker-instances.md)): + - when Portainer is enabled, the stored endpoint is created or updated with the name ` Portainer`, + the trimmed `BaseUrl` and the configured `EndpointId`, and marked enabled; + - when Portainer is disabled, an existing stored endpoint is kept but marked disabled; + - when the Docker instance is removed from configuration, its Portainer endpoint and action runs are deleted with it. +- Credentials (`Username`, `Password`, `ApiToken`) are never stored in the database; they are read from configuration + at the time of each call. +- `EndpointId` selects the Portainer environment that manages the Docker instance. When it is empty, the first + environment returned by Portainer's environment list is used, resolved again on every call. + +### Transport security + +- `BaseUrl` must be an absolute `https` URL. A plaintext `http` URL is accepted only with `AllowInsecureHttp: true`; + without it, startup fails with a message that names the key to set ([Hosting](hosting.md)). +- `AllowInsecureHttp` is opt-in only. It is a validation switch: no runtime warning is logged while it is active. +- Portainer server certificates are always validated; there is no option to skip validation for Portainer. + +### Authentication + +- When `ApiToken` is set, it is sent as an API key header on every request and takes precedence over username and + password. +- Otherwise `Username` and `Password` are posted to Portainer's authentication endpoint and the returned JWT is used as + a bearer token. The login is repeated for every operation; tokens are not cached. +- A login that answers non-success, or answers success without a token, ends the operation with an authentication + error that names the Portainer base URL and the HTTP status code; credentials never appear in it or in the log. + +### Capability check + +- For a Docker instance, a capability check reports whether Portainer is configured, whether actions are supported, and + a message: + - Portainer disabled or without `BaseUrl` → not configured, no actions; + - Portainer's status endpoint answers non-success → configured, no actions, message with the HTTP status code; + - no environment configured and none found → configured, no actions, "No Portainer endpoint found"; + - otherwise → configured, actions supported, message naming the environment used; + - any error (including authentication failure) → configured, no actions, message with the error text. + +### Actions + +- The executable actions are container actions only: `start`, `stop`, `restart`, `kill`, `pause` and `unpause` + (case-insensitive). Any other action name is rejected before Portainer is contacted, with a message listing the + allowed names. +- The target container is named, not identified by ID: Portainer's container list for the environment (including + stopped containers) is searched for an exact, case-insensitive name match, with or without the leading `/`. +- The action is executed by Portainer on the Docker Engine of that environment. The result is either *succeeded* with + a confirmation message or *failed* with a reason: Portainer not configured, no environment found, container not + found, HTTP status code of a non-success answer, or the error text. +- Actions never throw to the caller; every outcome is a result. Executed, failed and not-found actions are logged with + the action, the container name and the Docker instance name. +- Portainer requests go through the shared retry of transient HTTP failures described in + [Registries](registries.md). This includes the login and the action request itself. +- `RequestTimeoutSeconds` (1–300, default 15) bounds each Portainer request including its retries. + +### Action runs + +- The data model holds an action-run record per requested action: the Docker instance, optionally the Portainer + endpoint and the container snapshot, the action type (*restart container*, *redeploy stack*, *update service*), the + resource type (*container*, *service*, *stack*), the resource name (up to 200 characters), status (*requested*, + *running*, *succeeded*, *failed*, *cancelled*), requester, Portainer task ID, error message, and request and + completion times. +- A container snapshot referenced by an action run is never removed by retention ([Scan engine](scan-engine.md)). + Deleting the Portainer endpoint or the snapshot keeps the action run and clears the link. +- Today no component executes actions or writes action runs: neither a page nor a background job calls the capability + check or an action (see *Known gaps*). + +## Limits and errors + +- Validation, applied only when `Enabled` is `true`, fails startup when: `BaseUrl` is missing; `BaseUrl` is not an + absolute `http`/`https` URI; `BaseUrl` is `http` without `AllowInsecureHttp: true`; neither `ApiToken` nor both + `Username` and `Password` are set; `RequestTimeoutSeconds` is outside 1–300. +- A disabled Portainer section is not validated at all; its values may be incomplete. +- Error messages and log entries name the Docker instance, the Portainer base URL, the action, the container name and + HTTP status codes. They never contain the password, the API token or the JWT. + +### Known gaps + +- Container actions cannot be triggered from the UI or a job, and no action run is ever recorded. The executable + actions (`start`, `stop`, `restart`, `kill`, `pause`, `unpause` on containers) do not match the action types of the + action-run record, and redeploying stacks and updating services are not implemented. Tracked in + [#280](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/280). +- The login and the action request are retried on transient answers like any other request, so one requested action + can be executed more than once ([Registries](registries.md)). Tracked in + [#273](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/273). +- `AllowInsecureHttp: true` logs no runtime warning, unlike `SkipCertificateValidation` for Docker instances. Tracked + in [#276](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/276). +- Without `EndpointId` the first environment Portainer lists is used, which can be an environment other than the one + that manages the Docker instance; an explicit `EndpointId` is not required when Portainer manages several + environments. Tracked in [#279](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/279). + +## Related decisions + +None yet. + +## Not here + +- Container discovery, snapshots and Docker Engine connectivity: [Docker instances](docker-instances.md). +- The retry policy and the shared failure model of outbound HTTP: [Registries](registries.md). +- Startup validation failure and configuration reload: [Hosting](hosting.md). +- Retention of snapshots and other history: [Scan engine](scan-engine.md). +- What the UI shows about Portainer availability: [Web UI](web-ui.md). diff --git a/docs/areas/docker-instances.md b/docs/areas/docker-instances.md new file mode 100644 index 0000000..f1d9314 --- /dev/null +++ b/docs/areas/docker-instances.md @@ -0,0 +1,181 @@ +# Docker instances + +## Scope + +How DockerUpdateGuard connects to the Docker Engines it watches: the configured instances and their transports, the +read-only Engine calls it makes, the discovery of containers and their runtime status, the container snapshots that +record what ran where, and the CPU, memory and network samples collected per container and per instance. Operators +rely on it to configure engine access safely; the scan engine, update detection, base-image resolution and the UI rely +on the snapshots and samples it produces. + +## Behavior + +### Configured instances + +- Docker instances exist only through configuration: each entry of `DockerUpdateGuard:DockerInstances` is one + instance. There is no way to add an instance from the UI. +- Per instance the keys are `Name` (required), `BaseUrl` (required), `Enabled` (default `true`), `UseTls` (default + `false`), `SkipCertificateValidation` (default `false`), `CertificatePath` (optional), `RequestTimeoutSeconds` + (default `15`) and the optional `Portainer` section (see [Container actions](container-actions.md)). +- An instance is identified by its trimmed `Name`, compared case-insensitively. Renaming an instance in configuration + is the removal of the old instance and the creation of a new one. +- The configured instances are synchronized into the database at startup, on every + `DockerUpdateGuard:Scanning:DiscoveryIntervalMinutes` tick, and at the start of every runtime container scan and + every resource-statistics collection (schedules: [Scan engine](scan-engine.md)). A synchronization: + - creates a record for each new name, with registration source *configuration file*; + - updates endpoint URI (trimmed `BaseUrl`), connection kind, enabled flag and `SkipCertificateValidation` of each + existing one; + - **deletes** every stored instance whose name is no longer configured, immediately and without waiting for + retention, together with its scan runs, container snapshots, resource samples, Portainer endpoint and action + runs. A removed instance is not kept as disabled. Update findings that pointed at a deleted snapshot lose that + link but are kept. +- A disabled instance (`Enabled: false`) keeps its stored history but is not contacted: no discovery, no inspection, + no resource sampling. Every Engine call for it answers *not configured*. +- The stored connection kind is derived from `BaseUrl`: `http` → Http, `https` → Https, `tcp` → Https when `UseTls` + is `true`, otherwise Http, `unix` → UnixSocket, `npipe` → NamedPipe. Any other value is stored as *not set* and an + enabled instance with it is logged as a warning. + +### Transports + +- `http://` and `https://` URLs are used as the Engine base address as given. +- `tcp://host:port` is turned into `https://host:port` when `UseTls` is `true`, otherwise `http://host:port`. + `UseTls` has no effect on any other scheme. +- `unix:///path/to/socket` connects to that Unix domain socket (for example `unix:///var/run/docker.sock`). +- `npipe://server/pipe/name` connects to a Windows named pipe; an empty host or `.` means the local machine, a leading + `pipe/` segment is optional. A named-pipe URL without a pipe name fails every call for that instance. +- Engine connections never use an HTTP proxy. Connections are pooled per instance and per connection-relevant + setting (name, `BaseUrl`, TLS, certificate validation, certificate path); pooled TCP connections are renewed after + at most 5 minutes. +- Engine requests use the Docker Engine API version `v1.41` path prefix. +- Engine requests are not retried; a failed request fails that operation (the shared retry of outbound HTTP described + in [Registries](registries.md) does not apply to Engine connections). + +### TLS options + +- `SkipCertificateValidation: true` accepts any server certificate on `https` and TLS `tcp` connections. It is opt-in + only. Whenever it is active, a warning saying that the connection is unauthenticated and open to man-in-the-middle + attacks is logged once per instance name per process lifetime, when the first connection for that instance is + created. +- `CertificatePath` names a client certificate presented on `https` and TLS `tcp` connections. A `.pfx` or `.p12` + file is loaded as a password-less PKCS#12 bundle; any other extension is loaded as a single certificate file. A path + that does not exist is ignored without a warning. +- Neither option has an effect on `unix` and `npipe` connections. + +### Timeouts + +- `RequestTimeoutSeconds` (1–300, default 15) bounds each single Engine request (container list, each image inspect, + each image history, each stats call, host info) separately; it is not a budget for a whole scan. +- A request that exceeds it ends as *failed* with a message naming the instance and the timeout. Cancellation of the + calling job (for example at shutdown) is not reported as a failure; it stops the operation. + +### Container discovery + +- Discovery lists **all** containers of an instance, including stopped ones. +- Per container it records: the container ID; the name (the first listed name without its leading `/`, or the + container ID when it has no name); the image reference as the Engine reports it; the local image ID; the compose + project (`com.docker.compose.project` label), the stack name (`com.docker.stack.namespace`) and the swarm service + name (`com.docker.swarm.service.name`). +- The runtime status is one of Running, Paused, Restarting, Exited, Dead; any other Engine state (for example + `created`) is *not set*. A container counts as running only in state `running`. +- The image digest of each container is taken from the repository digests of its local image (one image inspect per + distinct local image): the digest whose registry and repository match the container's image reference; if none + matches, the only digest when there is exactly one; otherwise none. When the image reference cannot be parsed, the + first repository digest is used. Digests are stored lower-case. A failed inspect leaves the digest empty and does + not fail discovery. +- Discovery emits one `dockerengine.request` trace activity per instance (see [Telemetry](telemetry.md)). + +### Image inspection and history + +- Other areas can read the inspect data of a local image (ID, repo tags, repo digests, environment variables, root + file-system layers, creation time, operating system, architecture) and its history (creation time, created-by + command, comment, tags per layer) from an instance, addressed by image ID or reference. +- An image the Engine does not know answers *not found*; any other non-success answer *failed*. How the data is used + belongs to [Base images](base-images.md) and [Update detection](update-detection.md). + +### Container snapshots + +- Each runtime container scan of an enabled instance records one scan run for the instance and one container snapshot + per discovered container, linked to that scan run. Snapshots are append-only history: every scan adds new rows, + earlier ones stay until retention removes them ([Scan engine](scan-engine.md)). +- A snapshot holds the instance, container ID, name, compose project, stack name, service name, runtime status, + running flag, the catalogued image version (registry, repository, tag and the discovered digest, see + [Image catalog](image-catalog.md)) and its recording time. It starts with update assessment *not evaluated*; the + assessment fields are filled by [Update detection](update-detection.md). +- The **current** containers of an instance are the snapshots of its most recent runtime scan run that recorded + snapshots. A later run that recorded none (for example because the Engine was unreachable) does not empty the + current set; the previous set stays current. +- A container whose image reference cannot be parsed produces no snapshot; the scan run then ends *partial* with a + message naming the container. +- Outcome of the discovery step for the scan run: a *failed* discovery ends the run *failed*; *not configured* or + *unsupported* ends it *partial*; an unexpected error ends it *failed*. The run's error message joins the distinct + messages of all problems. One failing instance never stops the scans of the other instances. +- Configured instances that are disabled, or stored instances that are missing from the current configuration, are + skipped by the scan without a scan run. + +### Resource statistics + +- Every `DockerUpdateGuard:Scanning:ResourceStatisticsIntervalMinutes` (1–1440, default 5) each enabled, configured + instance is sampled ([Scan engine](scan-engine.md) owns the schedule). +- Only **running** containers are sampled, one at a time, each with a single non-streaming stats request. A container + whose stats request answers non-success is left out of that round. +- Per container sample: + - CPU percent = (container CPU time delta ÷ system CPU time delta) × online CPUs × 100, rounded to 4 decimals; the + CPU count falls back to the per-CPU usage list length and then to 1; 0 when either delta is not positive. A + multi-core container can exceed 100. + - Memory usage and memory limit are the Engine's reported usage and limit in bytes. + - Network receive and transmit totals are the sums over all of the container's network interfaces, in bytes. + - The recording time is the Engine's stats timestamp, or the current time when it is missing. + - Receive and transmit rates in bytes per second are computed against the previous sample of the same container ID + on the same instance, rounded to 4 decimals. The rate is 0 for the first sample, when no time has passed, or when + the total did not grow (for example after a container restart reset the counter). +- Per instance, one sample per round: container count, CPU percent and memory usage as the sums over the container + samples, memory limit as the host's total memory reported by the Engine (0 when that request fails), network totals + as sums, rates against the previous instance sample by the same rule, and the latest container recording time (the + current time when no container was sampled). An instance with no running containers still gets a sample with + count 0. +- Samples are kept until retention removes them ([Scan engine](scan-engine.md)); how they are charted belongs to + [Web UI](web-ui.md). + +## Limits and errors + +- Invalid instance configuration fails startup ([Hosting](hosting.md)): a missing `Name`, a duplicate name (after + trimming, case-insensitive), a missing `BaseUrl`, a `BaseUrl` that is not an absolute URI with scheme `http`, + `https`, `tcp`, `unix` or `npipe`, or a `RequestTimeoutSeconds` outside 1–300. The shipped `appsettings.json` + contains one instance with an empty `Name`, so an operator must set a name before the application starts. +- Every Engine operation reports its outcome as a status (*succeeded*, *not configured*, *unsupported*, *not found*, + *failed*) instead of throwing; the shared failure model is described in [Registries](registries.md). +- A non-success Engine answer is reported as *failed* with the instance name, the HTTP status code and the response + body. Unexpected errors are reported with the instance name and the error message. These messages can end up in a + scan run's error message and are shown in the UI. +- A resource collection that fails for an instance (list request failed or timed out, or any stats request timed out + or broke) records **no** samples for that instance in that round and logs a warning; other instances are not + affected. +- `SkipCertificateValidation` and a mounted Docker socket are trust relaxations the operator chooses; the socket is + root-equivalent on the host ([Hosting](hosting.md)). + +### Known gaps + +- Instance lookups during a runtime scan or a resource collection compare the configured `Name` untrimmed with the + trimmed stored name, so an instance whose configured name has leading or trailing spaces is stored but always + skipped as missing from configuration. Tracked in + [#266](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/266). +- A `CertificatePath` that is not a `.pfx`/`.p12` file is loaded as a certificate without a private key, which cannot + authenticate a TLS client, and a `CertificatePath` file that does not exist is ignored without validation error or + warning. Tracked in [#269](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/269). +- A single timed-out or broken stats request discards the whole round of samples for its instance, while a + non-success stats answer only drops that container. Tracked in + [#270](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/270). + +## Related decisions + +None yet. + +## Not here + +- When scans and sampling run, retention of snapshots, samples and scan runs: [Scan engine](scan-engine.md). +- How a container's image is evaluated for updates and the assessment statuses: [Update detection](update-detection.md). +- Base-image and derived-runtime detection from inspect and history data: [Base images](base-images.md). +- Image reference parsing and the image catalog: [Image catalog](image-catalog.md). +- Restart and other actions through Portainer: [Container actions](container-actions.md). +- Pages that show instances, containers and resource charts: [Web UI](web-ui.md). +- Startup, validation failure and the container image's socket access: [Hosting](hosting.md). diff --git a/docs/areas/hosting.md b/docs/areas/hosting.md new file mode 100644 index 0000000..7994ea9 --- /dev/null +++ b/docs/areas/hosting.md @@ -0,0 +1,191 @@ +# Hosting and configuration + +## Scope + +How the application is configured, how it starts, how it serves HTTP and how its container image is built to run. +It covers loading and validating configuration, resolving the database connection, the startup sequence up to the +point where requests are served, configuration reload, the HTTP-level pipeline and error pages, the container image +and the security posture the deployment must provide. Operators rely on it to deploy and configure the application; +every feature area relies on it for validated configuration and a migrated database. + +## Behavior + +### Configuration sources + +- Configuration is read from, in increasing precedence: `appsettings.json`, `appsettings.{Environment}.json`, + environment variables and command-line arguments. Environment variables use `__` as the + section separator (`DockerUpdateGuard__Scanning__RetryCount`). +- The application reads four top-level sections: `DockerUpdateGuard` (application settings), `Telemetry` (see + [Telemetry](telemetry.md)), `ConnectionStrings` and `Logging`. The key-by-key reference with defaults and ranges is + the configuration reference in `README.md`; what each feature key means is stated in the feature's area document. +- The environment defaults to `Production`. Only `Development` changes behavior here (see *HTTP pipeline*). +- Secrets (database password, Docker Hub PAT, Portainer password or API token) are ordinary configuration values; + any of the sources above may supply them. There is no further secret-store integration. + +### Validation + +- The whole `DockerUpdateGuard` section is validated as one unit. All rule violations are collected and reported + together, each naming the offending key (for example `'DockerUpdateGuard:Database:MigrationRetryDelaySeconds' must be + greater than zero`). Messages name keys and, at most, configured instance names; they never echo connection + strings, passwords or tokens. +- Any violation fails startup: the process exits before the database is contacted and before any request is served. +- The `Telemetry` section is validated even earlier, while services are registered; see [Telemetry](telemetry.md). +- Rules owned by this area: + - a connection string must resolve to a non-blank value (see *Database connection*); + - `DockerUpdateGuard:Database:MaxConnectionRetryCount`, `MaxConnectionRetryDelaySeconds`, + `MigrationStartupTimeoutSeconds` and `MigrationRetryDelaySeconds` must each be greater than zero. +- The rules for feature keys (Docker Hub, release metadata, vulnerabilities, scanning, Docker instances, Portainer) + are part of the respective area documents; they are enforced by the same validation step. + +### Database connection + +- The database is PostgreSQL. The connection string is resolved in this order: + 1. `DockerUpdateGuard:ConnectionString`, trimmed, when it is not blank; + 2. otherwise `ConnectionStrings:{name}`, where `{name}` is `DockerUpdateGuard:ConnectionStringName` + (default `DockerUpdateGuard`). +- Validation and the database registration use the same resolution, so the validated connection string is always + the one in use. +- During normal operation, a database command that fails transiently is retried up to + `DockerUpdateGuard:Database:MaxConnectionRetryCount` times (default `8`) with a back-off of at most + `DockerUpdateGuard:Database:MaxConnectionRetryDelaySeconds` seconds (default `30`) between attempts. + +### Startup sequence + +Startup runs these steps in order; requests are served and background jobs start only after all of them have +succeeded. + +1. **Configuration** is bound and validated (see *Validation*). +2. **Wait for the database.** The application opens and closes a connection. A transient failure (a database error + the driver reports as transient, a socket error or a timeout, also when wrapped in another error) is retried every + `DockerUpdateGuard:Database:MigrationRetryDelaySeconds` seconds (default `5`) until + `DockerUpdateGuard:Database:MigrationStartupTimeoutSeconds` (default `120`) have elapsed since the first attempt. + Each retry is logged with its attempt number. A non-transient failure, or a transient one after the deadline, is + logged and fails startup. A shutdown request during the wait ends it. +3. **Apply migrations.** Pending schema migrations are applied automatically. On PostgreSQL the application first + takes the session-scoped advisory lock with key `5410320540103205` on the same connection that runs the + migrations, so instances starting concurrently against one database apply migrations one at a time; the others + wait for the lock and then find nothing left to apply. Taking the lock and migrating form one unit: if a transient + fault forces a retry, both are repeated together. The lock is released afterwards; a failed release is logged and + does not change the outcome, because the lock also ends with the session. A failed migration is logged and fails + startup. +4. **Synchronize configured Docker instances** into the database (see [Docker instances](docker-instances.md)). +5. **Seed the inventory gauges** from the database (see [Telemetry](telemetry.md)). + +Only local database work runs on this path. Work that needs the network, such as Docker Hub account image +discovery, runs in the background jobs after startup (see [Scan engine](scan-engine.md)), so an unreachable +registry never delays or prevents startup. + +### Configuration reload + +- `appsettings.json` and `appsettings.{Environment}.json` are watched; a change is picked up without a restart. + Values that are read on use take effect at their next use; the feature areas state which of their keys do, and the + [Scan engine](scan-engine.md) states when job intervals change. +- These values are read once at startup and need a restart to change: + - the connection string, `DockerUpdateGuard:ConnectionStringName` and all `DockerUpdateGuard:Database:*` keys; + - `DockerUpdateGuard:DockerHub:ApiBaseUrl`, `DockerUpdateGuard:DockerHub:RequestTimeoutSeconds` (also the request + timeout for generic registries and release-metadata feeds) and `DockerUpdateGuard:ReleaseMetadata:*`; + - the whole `Telemetry` section. +- A reloaded configuration is validated by the same rules as at startup when it is next read; a reload that fails + validation raises an error where it is read (see *Known gaps*). + +### HTTP pipeline + +- The application serves only its interactive web UI and static assets; there is no separate HTTP API. +- Outside `Development`: + - an unhandled exception while processing a request renders the `/error` page with a generic message + ("Unexpected error"); no exception details, stack traces or configuration values are shown; + - HSTS is applied and HTTP requests are redirected to HTTPS when an HTTPS port is known to the server. +- In `Development`, the platform's developer exception page is shown instead and there is no HSTS or HTTPS + redirection. +- In every environment, a response with an error status code and no body (such as `404`) is re-rendered as the + `/not-found` page, and an unknown route inside the UI shows the same page. Both pages are described in + [Web UI](web-ui.md). +- Form posts are protected by anti-forgery validation. + +### Container image + +- The runtime image is based on the Alpine variant of the ASP.NET Core runtime (`mcr.microsoft.com/dotnet/aspnet`, + `10.0-alpine` by default). The base image name and digest are recorded as the OCI labels + `org.opencontainers.image.base.name` and `org.opencontainers.image.base.digest`. +- The application listens on plain HTTP on port `8080` on all interfaces. TLS is not terminated inside the container. +- The process runs as UID `64000` in the root group (GID `0`), never as UID `0`. `/app` is owned by `64000:0` with + group permissions equal to owner permissions, so the application also works when started with another UID in + group `0`. Membership in group `0` keeps a `root:root` Docker socket readable where the socket grants group access. +- The Trivy CLI is bundled at `/usr/local/bin/trivy` (taken from `aquasec/trivy:0.58.1` by default) with its cache in + `/tmp/trivy`, so Trivy client mode works without extra installation (see [Vulnerabilities](vulnerabilities.md)). +- The image sets `DockerUpdateGuard__DisplayVersion` to the version it was built with. +- CA certificates are installed so outbound TLS connections can be verified. +- Before the application starts, the entrypoint: + 1. checks that `/app/appsettings.json` and `/app/appsettings.{Environment}.json`, if present, are readable by the + running user; if one exists but is not readable, it prints the UID/GID and how to fix the host permissions, + and exits with code `1`; + 2. if `/app/certs` contains files, imports every `.crt` and `.pem` file into the system trust store, but only when + running as root or when the trust store is writable; otherwise, or when the import fails, it prints a message + and continues. Certificate import never stops the container. + +### Security posture + +These properties hold today and must be provided for by the deployment: + +- **No authentication or authorization.** Every page and every action (manual scans, tag selection, container + actions) is available to anyone who can reach port `8080`. The application must run behind a trusted perimeter: a + private network, a VPN or a reverse proxy that authenticates users. +- **No health-check endpoint and no rate limiting.** +- **Docker socket access is root-equivalent on the host.** Mounting the host's Docker socket gives the application, + and anyone who controls it, the power of root on that host. This is inherent to the feature. +- **Single instance.** Several instances may share one database for migrations (see *Startup sequence*), but the + background jobs do not coordinate across instances (see [Scan engine](scan-engine.md)). +- That secrets never appear in logs or telemetry is stated in [Telemetry](telemetry.md); that the UI never shows + them is stated in [Web UI](web-ui.md). + +## Limits and errors + +| Situation | Result | +| --- | --- | +| Invalid `DockerUpdateGuard` configuration at startup | Startup fails; every violation is reported with its key. | +| No connection string resolves | Startup fails: `'DockerUpdateGuard:ConnectionString' or 'ConnectionStrings:{name}' must be configured`. | +| Invalid `Telemetry` configuration | Startup fails before the host is built (see [Telemetry](telemetry.md)). | +| Database unreachable within `MigrationStartupTimeoutSeconds` | The attempts are logged; startup fails. | +| Non-transient database error while waiting | Startup fails immediately. | +| Migration fails | Logged; startup fails. | +| Advisory lock cannot be released | Logged; the migration outcome stands. | +| Mounted settings file not readable by the container user | Entrypoint exits with code `1` and a fix-it message. | +| Certificate import not possible | A message is printed; the container starts anyway. | +| Unhandled exception in a request (non-Development) | The `/error` page with a generic message. | +| Unknown URL or bodyless error status | The `/not-found` page. | + +### Known gaps + +- An invalid configuration reload is not rejected while the last valid configuration stays in use: the background + jobs read their interval from the reloaded configuration outside their error handling, so a reload that fails + validation can stop the host ([Scan engine](scan-engine.md)). + Tracked in [#281](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/281). +- Forwarded headers from a reverse proxy are not supported. Behind a TLS-terminating proxy the application sees + plain HTTP, so it never sends HSTS and HTTPS redirection cannot determine a port; both are left to the proxy. + Tracked in [#283](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/283). +- Certificates in `/app/certs` are not imported for the default user: in the shipped image the system trust store is + owned by root and not writable by group `0`, so the import is skipped unless the container runs as root. + Tracked in [#285](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/285). +- Waiting for the migration advisory lock is bounded by the 30-second database command timeout and then retried like + any transient database failure (see *Database connection*), so an instance that starts while another one runs a + long migration can fail its startup after several minutes. + Tracked in [#287](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/287). +- The shipped `appsettings.json` configures one Docker instance with an empty name, which fails configuration + validation; the application does not start with the shipped settings until that instance is named or removed. + Tracked in [#288](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/288). + +## Related decisions + +None yet. + +## Not here + +- What each feature key means and its valid range: the feature areas ([Docker instances](docker-instances.md), + [Registries](registries.md), [Base images](base-images.md), [Update detection](update-detection.md), + [Vulnerabilities](vulnerabilities.md), [Container actions](container-actions.md), [Scan engine](scan-engine.md)). +- The complete configuration key reference: `README.md`, *Configuration reference*. +- Retry of outbound HTTP calls: [Registries](registries.md). +- Logs, traces, metrics and the rule that credentials stay out of them: [Telemetry](telemetry.md). +- Pages and what they show: [Web UI](web-ui.md). +- Building, testing, CI and releasing the image: `docs/CONTRIBUTING.md`. +- Compose files and the walkthrough for the deployment tiers: `INSTALL.md`. diff --git a/docs/areas/image-catalog.md b/docs/areas/image-catalog.md new file mode 100644 index 0000000..dc19e8d --- /dev/null +++ b/docs/areas/image-catalog.md @@ -0,0 +1,121 @@ +# Image catalog + +## Scope + +What "an image" is in DockerUpdateGuard: how image references are parsed and normalized, the deduplicated catalog of +registry repositories and image versions (repository, tag and digest) that every other area points at, how the catalog +stays free of duplicates under concurrent writers, and observed images — the images a user registers or the Docker Hub +account discovery adds to be tracked, with their source and enabled state. Every scanning area and the UI rely on it to +refer to the same image the same way. + +## Behavior + +### Image references + +- A reference has the form `[registry/]repository[:tag][@digest]`; surrounding whitespace is ignored. +- **Digest:** everything after the first `@`, trimmed and lowercased. +- **Tag:** the text after the last `:` that follows the last `/`, trimmed, with its case kept. A missing or empty tag + is `latest`, also for a reference that names only a digest. +- **Registry:** the first path segment, lowercased, when the reference has at least two segments and that segment + contains a `.` or a `:` or equals `localhost` (case-insensitive). Otherwise the registry is `docker.io`. +- **Repository:** the remaining segments joined with `/`, lowercased; empty segments are ignored. A reference with a + single segment gets the `library/` namespace (`nginx` → `docker.io/library/nginx:latest`). +- **Mirror-wrapped references:** when the registry is `docker.io` and the first repository segment is + `mcr.microsoft.com`, `ghcr.io`, `quay.io` or a host-like name containing `harbor`, that segment becomes the registry + (`docker.io/ghcr.io/owner/app:1` → `ghcr.io/owner/app:1`). +- No other validation happens when parsing: characters and lengths are not checked. A reference without a repository + name is rejected. +- Docker Hub spellings are not unified: `docker.io/nginx` has the repository `nginx`, not `library/nginx`, and + `index.docker.io/…` is a different registry from `docker.io/…`. Registry access still treats them alike + ([Registries](registries.md)), but they are separate catalog entries. +- The canonical text form is `registry/repository:tag`, followed by `@digest` when the digest is known. Parsing the + canonical form gives back the same parts. + +### The catalog + +- A **registry repository** is a unique pair of registry (at most 200 characters) and repository (at most 300). +- An **image version** is a unique combination of registry repository, tag (at most 200 characters) and digest (at most + 255). The digest is optional; an image version without one is a different entry from the same tag with a digest. +- Registry, repository and tag must be non-blank. The catalog trims the digest and drops any `name@` prefix in front of + it, but changes no case: parts are compared exactly, so writers pass references normalized as above. +- Writers use get-or-create: an existing entry is returned unchanged; a new one stores the given publication time and + metadata (at most 4000 characters). Refreshing the publication time or metadata of an existing entry is done by the + component that owns the refresh ([Base images](base-images.md) for observed images and their bases, the runtime + container scan for container images, see [Update detection](update-detection.md)). +- **Concurrency:** when two writers create the same entry at once, the one whose insert collides with the uniqueness + rule re-reads and returns the entry the other created. No duplicates arise; no writer sees an error for the race. +- Every image version records which kind of component registered it last: observed image, runtime container or + base-image resolution. The value is informational and is overwritten by the next registering component. +- Catalog entries are shared: observed images, container snapshots, base-image relationships, update findings and + vulnerability findings all point at the same entries. Catalog entries are never deleted. + +### Live image versions + +- The live image versions are: the current version of every observed image (enabled or not); the image versions of + the containers in the latest runtime scan of each Docker instance (for an instance without a runtime scan, the + latest snapshot of each of its containers); and the base images recorded for any of these versions. +- The live set decides which images get vulnerability refreshes and which findings the cleanup keeps + ([Vulnerabilities](vulnerabilities.md), [Scan engine](scan-engine.md)). + +### Observed images + +- An observed image is a tracked target: a name (at most 200 characters, not required to be unique), an optional + description, its current image version, an enabled flag, its source and its creation and update times. +- Sources: *manual* (registered on `/observed-images`) and *discovery* (added by the Docker Hub account discovery and + shown as "My images" on `/my-images`; see [Registries](registries.md)). Discovered images are the user's own images. +- **Manual registration** takes a name and an image reference (both required) and an optional description, then: + 1. parses the reference and gets or creates its image version exactly as written — tag and, if given, digest — with + no registry call; + 2. looks for an observed image whose stored name equals the entered name exactly (case-sensitive, untrimmed); + 3. creates a new observed image (name trimmed, description trimmed or none, source *manual*, enabled), or updates + the found one: description replaced, current version replaced, enabled again; name and source stay; + 4. starts a manual scan of that image ([Base images](base-images.md)) and opens its detail page. +- The current version of a manual image never moves on its own: scans refresh its metadata but keep its tag and + digest. A discovered image's current version is moved by each discovery run to the repository's most recently + pushed tag. +- **Enabled state:** only discovery disables an observed image (when its repository disappears from the account) and + it enables again every image it lists. Manual registration of an existing name enables it again. There is no way to + disable or delete an observed image from the UI. Disabled images are skipped by the scheduled observed-image scan + ([Scan engine](scan-engine.md)) and keep their history. +- **Own images and running containers:** a running container is linked to an own image when its image's registry and + repository equal (case-insensitive) those of a discovered image's current version, regardless of tag and digest; + when several match, the one first by name wins. Manually registered images are not linked this way. + +## Limits and errors + +- A blank name or reference is rejected by the registration form. A reference without a repository name, or a value + longer than the stored limits, makes the registration fail; the form shows the error message and the observed image + is not saved (an image version created before the failure remains in the catalog). +- A name entered with leading or trailing whitespace does not match the stored, trimmed name and creates a further + observed image. +- The catalog rejects a blank registry, repository or tag. +- Nothing in this area handles credentials. + +### Known gaps + +- Docker Hub references are not normalized to one form (`docker.io/nginx` vs `nginx`, `index.docker.io` vs + `docker.io`), so one image can have several catalog entries and is then not recognized as shared. Tracked in + [#275](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/275). +- Manual registration identifies an existing observed image by its untrimmed, case-sensitive name, and names are not + unique: near-identical names create separate entries, and registering a manual image under a discovered image's name + updates the discovered one, which the next discovery run overwrites. + Tracked in [#284](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/284). +- Observed images cannot be disabled or deleted by the user, and discovery enables every listed repository again on + each run. Tracked in [#282](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/282). +- The recorded kind of the component that last registered an image version is last-writer-wins and carries no + meaning, and a further kind (registry metadata) is defined but never written. + Tracked in [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). + +## Related decisions + +None yet. + +## Not here + +- Reading tags and metadata from registries and the account discovery itself: [Registries](registries.md). +- Base-image chains and derived runtimes: [Base images](base-images.md). +- Comparing tags and choosing updates: [Update detection](update-detection.md). +- Container snapshots: [Docker instances](docker-instances.md). +- Vulnerability findings: [Vulnerabilities](vulnerabilities.md). +- Scan runs, schedules and retention: [Scan engine](scan-engine.md). +- What the pages show: [Web UI](web-ui.md). diff --git a/docs/areas/registries.md b/docs/areas/registries.md new file mode 100644 index 0000000..72dbaa4 --- /dev/null +++ b/docs/areas/registries.md @@ -0,0 +1,205 @@ +# Registries + +## Scope + +Read access to container registries: Docker Hub (its web API and its registry) and every other registry that speaks the +OCI Distribution API. Covers which client serves which registry, what can be read (tags, manifests, image +configuration and labels), how DockerUpdateGuard authenticates, the Docker Hub account image discovery, the Docker Hub +request budget and parallelism, the retry of transient HTTP failures shared by the outbound HTTP integrations, and the +failure model every integration uses. The image catalog, base-image resolution and update detection rely on it for +registry data; operators rely on it to configure credentials and limits; the other integration areas link here for the +retry policy and the failure model. + +## Behavior + +### Failure model of integrations + +- An integration call answers with an outcome instead of raising an error for an expected failure. The outcomes are: + - *succeeded* — the call returned data; + - *not configured* — the feature lacks the configuration it needs; + - *unsupported* — the integration cannot serve this target (for example a registry no client handles); + - *not found* — the target does not exist (HTTP 404 for an account, repository, tag, manifest or blob); + - *failed* — the call was made and did not succeed (any other non-success status, a missing token, a response + without the expected content); + - *unknown* — the outcome could not be classified. +- Every outcome other than *succeeded* carries a human-readable message. Callers decide on the outcome, not on errors; + how a scan turns outcomes into scan-run statuses is described with each scan + ([Base images](base-images.md), [Update detection](update-detection.md), [Scan engine](scan-engine.md)). +- A *failed* outcome from a registry names the HTTP status code and includes the response body the remote side sent. + +### Choice of client per registry + +- The registry of an image reference (see [Image catalog](image-catalog.md)) decides which client serves it. The Docker + Hub client serves the hosts `docker.io`, `index.docker.io`, `registry-1.docker.io`, `hub.docker.com`, `docker.com` + and `www.docker.com`, compared case-insensitively; a registry given as an absolute URL is matched by its host. +- Every other non-empty registry is served by the generic OCI client. Only an empty registry is *unsupported*. +- The Docker Hub client uses three endpoints: the web API at `DockerUpdateGuard:DockerHub:ApiBaseUrl` (default + `https://hub.docker.com/`), the token service `https://auth.docker.io/token` and the registry + `https://registry-1.docker.io`. A repository without a namespace is addressed as `library/`. +- The OCI client addresses `https:///v2//…`. A registry given as an absolute URL is used with its + own scheme and base path; registry names are compared lowercased and keep a non-default port. +- Outbound requests to registries and release feeds carry the user agent `DockerUpdateGuard/1.0`. + +### What can be read + +- **Tag lookup** — digest and publication time of one tag. + - Docker Hub: the web API's tag record (the digest Docker Hub reports and its `last_pushed` time). + - Other registries: a manifest request for the tag. The digest is the `Docker-Content-Digest` of the first response + (for a multi-platform index, the index digest). The publication time is known only when the manifest carries a + top-level `created` timestamp; otherwise it is unknown. +- **Image configuration** — environment variables, labels (keys compared case-insensitively), creation time, operating + system and architecture, read from the image's configuration blob. For a multi-platform index one platform manifest + is chosen: + - Docker Hub: the first `linux` entry, skipping entries whose operating system or architecture is `unknown` + (attestations); otherwise the first remaining entry. + - Other registries: the entry matching the caller's operating system and architecture, then the first entry matching + the architecture, then the first entry with a digest. A caller that passes no platform gets the first entry. +- **Tag listing** — the tags of a repository with digest and publication time. + - Docker Hub: web API pages of 100 tags, in the order Docker Hub returns them (most recently pushed first), following + the `next` link of each page. + - Other registries: `tags/list` pages of 100 names following the `Link: rel="next"` header; every tag is then + inspected with a tag lookup. A tag whose lookup does not succeed is still listed, by name only. + +### Bounded tag listings + +- A caller may bound a listing with the current tag, the current digest, a minimum version tag, a version-line tag, a + published-since time and a maximum count (default 250 when bounds are given). Without bounds every tag is read. +- The current tag is always kept. With a version-line tag only tags of that version line are kept (version lines and + version comparison: [Update detection](update-detection.md)). A tag published before the published-since time, or + lower than the minimum version, is dropped unless its digest equals the current digest; a tag that cannot be + compared with the minimum version is kept. +- Docker Hub: the listing stops once the maximum count of kept tags is reached, and at the first tag older than the + published-since time — except when the current tag is `latest` with a known digest and no specific version tag + with that digest has been seen yet. +- Other registries: the maximum count limits the number of inspected tags; a tag that fails the name checks (current + tag, version line, minimum version) is not inspected. + +### Authentication + +- **Docker Hub web API.** Anonymous unless `DockerUpdateGuard:DockerHub:Pat` is set. With `Pat` and `UserName` (both + trimmed) the client logs in and sends the returned token as a bearer token. The token is kept in memory and renewed + one minute before its expiry (its `exp` claim, or five minutes after issue when it has none). `Pat` without + `UserName` makes every web API request *failed*, naming the missing key. `UserName` without `Pat` is anonymous. +- **Docker Hub registry.** A pull token is requested from the token service for every lookup, with the configured user + name and PAT as basic credentials when both are set, otherwise anonymously. +- **Other registries.** Access is anonymous only. On a 401 with a `Bearer` challenge a token is requested anonymously + from the challenge's realm, with its `service` and `scope` (default `repository::pull`). Tokens are + cached per registry host and repository until `issued_at` + `expires_in` (60 seconds when absent) minus 5 seconds; a + 401 with a cached token drops it and authenticates again once. Any other challenge, or a failed token request, makes + the call *failed* — private repositories outside Docker Hub cannot be read. + +### Docker Hub account discovery + +- Discovers the repositories of the configured Docker Hub account as own images. It runs only when both `UserName` + and `Pat` are set; otherwise each run is skipped with a log entry. It runs every + `DockerUpdateGuard:Scanning:DockerHubAccountDiscoveryIntervalMinutes` ([Scan engine](scan-engine.md)). +- The account listing requires `DockerUpdateGuard:DockerHub:Registry` to name a Docker Hub host; otherwise it answers + *unsupported* and the run ends without changes. +- The account is the namespace equal to `UserName`. All its repositories are listed (pages of 100), including private + ones the token can see. Repository names are lowercased; duplicates collapse to the most recently updated one. +- For each repository the full tag list is read and one tag is tracked: the most recently pushed; on equal push time + `latest` first, then alphabetical; tags without a push time come last. +- Each repository becomes an observed image with source *discovery*, matched by registry and repository + (case-insensitive). A new one is created enabled; an existing one gets its name set to the repository, its + description set to the Docker Hub description, its current version set to the tracked tag, and is enabled again. + The catalog entry is created as described in [Image catalog](image-catalog.md), with the tag's digest and push time. +- Discovered observed images whose repository is no longer listed are disabled, never deleted. +- A repository whose tag listing does not succeed or has no tags is skipped; its observed image keeps its state. When + the account listing does not succeed, nothing changes. +- All changes of a run are saved together at its end. Discovery does not scan; discovered images are scanned by the + observed-image scan ([Scan engine](scan-engine.md)). + +### Docker Hub request budget and parallelism + +- `Scanning:DockerHubRequestLimitWindowHours` (default 6, range 1–168), `Scanning:DockerHubRequestLimitPerWindow` + (default 200, range 1–100000) and `Scanning:DockerHubReservedManualRequestsPerWindow` (default 40, range 0–100000, + smaller than the per-window limit) describe the Docker Hub request budget. The budget is a scheduling hint: it + assumes one Docker Hub request per observed image and scan and only stretches the schedule of the observed-image + scan ([Scan engine](scan-engine.md)). It does not count requests; a scan may issue several requests per image + (token, manifests, configuration blobs, tag lookups and listings), manual scans are not limited, and no request is + refused for budget reasons. +- `DockerUpdateGuard:DockerHub:MaxParallelRequests` (default 4, range 1–32) caps the requests one Docker Hub client + instance has in flight at once, covering web API, login, token and registry requests. Every scan and every UI + operation works with its own client instances, so the cap is not global. The value is read when an instance is + created, so a configuration reload applies to later operations. Requests to other registries are not capped. +- Docker Hub rate limiting (HTTP 429) is handled by the retry below. + +### Retry of transient HTTP failures + +- Applies to Docker Hub (web API, login, token service, registry), other registries and their token services, the + .NET and nginx release feeds ([Base images](base-images.md)) and Portainer + ([Container actions](container-actions.md)). It does not apply to Docker Engine connections + ([Docker instances](docker-instances.md)) or to Docker Scout ([Vulnerabilities](vulnerabilities.md)). +- Transient are the responses 408, 429 and every 5xx, and transport errors and timeouts raised while sending. Other + statuses (including 401, 403 and 404) and cancellation are not retried. +- A request is retried up to `DockerUpdateGuard:Scanning:RetryCount` times (default 2, range 0–10; 0 disables + retries). The value is read for every request, so a configuration reload applies at once. +- The wait before a retry is the response's `Retry-After` (delta or date) when positive, capped at 30 seconds; + otherwise 500 ms × 2^n for the n-th retry (n starting at 0), capped at 30 seconds, plus a random jitter of up to 20 % + of that value, the total again capped at 30 seconds. +- Every method is retried, including requests with a body (the body is buffered so it can be resent). +- When the retries are used up the last response is returned to the caller, or the last error is raised. +- Each retry is logged as a warning with the method, the URL without its query, the status or error, the attempt + number and the wait. +- `DockerUpdateGuard:DockerHub:RequestTimeoutSeconds` (default 30, range 1–300) is the total time one request may + take including all retries and waits, for Docker Hub, other registries and the release feeds. A request that runs + out of time is not retried. + +### Configuration reload + +- Read live: `DockerHub:UserName`, `DockerHub:Pat`, `DockerHub:Registry` (for account discovery), + `DockerHub:MaxParallelRequests` (for new client instances) and `Scanning:RetryCount`. +- Fixed at startup, a change needs a restart: `DockerHub:ApiBaseUrl` and `DockerHub:RequestTimeoutSeconds`. +- Validation of these keys at startup (an invalid value fails startup, see [Hosting](hosting.md)): `Registry` + non-empty, `ApiBaseUrl` an absolute `http`/`https` URL, and the ranges above. + +## Limits and errors + +- *Unsupported*: an empty registry; the account listing when `DockerHub:Registry` is not a Docker Hub host. +- *Not found*: HTTP 404 for an account, repository, tag, manifest or configuration blob; an image whose manifest names + no configuration blob. +- *Failed*: any other non-success status, a token that cannot be obtained, a PAT without user name, a multi-platform + index without a usable entry. The message contains the status code and the remote response body. +- Some failures still surface as errors instead of outcomes: transport errors and timeouts that remain after the + retries, and malformed JSON, during tag lookups, tag listings and the account listing (both clients) and during + image configuration lookups on other registries. A Docker Hub configuration lookup turns them into *failed*. The + scan that made the call records such an error as a failed scan. +- The PAT is never written to logs or messages. The user name appears in the log entry and the message of a failed + Docker Hub login. +- Registry responses are untrusted: missing or mistyped JSON fields are treated as unknown values. An unbounded + listing (account discovery) reads every page the registry offers. + +### Known gaps + +- `MaxParallelRequests` caps each Docker Hub client instance, not the process: concurrent scans, and the client + instances one scan uses, each get the full allowance. Tracked in + [#265](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/265). +- Non-idempotent requests with a body, such as the Docker Hub login and Portainer actions, are retried like any other + request, so one request can take effect more than once. Tracked in + [#273](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/273). +- `DockerHub:Registry` does not decide which registries the Docker Hub client serves (that host list is fixed); it + only gates the account listing and, when set to an absolute URL with a non-Docker-Hub host, replaces the web API + base address. Tracked in [#268](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/268). +- Transport errors, timeouts and malformed JSON escape as errors from several calls instead of being answered as + *failed* outcomes. Tracked in [#272](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/272). +- For other registries an image configuration lookup without a platform takes the first index entry, which can be an + attestation manifest, and publication times come only from a top-level `created` field of the manifest, which most + registries do not send, so their tags usually have no publication time. Tracked in + [#277](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/277). +- Pagination links (`next` on Docker Hub, `Link` elsewhere) are followed to whatever host they name, and the Docker + Hub bearer token is sent along to that host. Tracked in + [#264](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/264). + +## Related decisions + +None yet. + +## Not here + +- When scans and discovery run, and the formula that turns the request budget into a schedule: + [Scan engine](scan-engine.md). +- Parsing image references and storing what was read: [Image catalog](image-catalog.md). +- Following base-image labels and the release feeds' content: [Base images](base-images.md). +- Version lines, tag ranking and update recommendations: [Update detection](update-detection.md). +- Docker Scout and Trivy: [Vulnerabilities](vulnerabilities.md). +- Docker Engine access: [Docker instances](docker-instances.md); Portainer: [Container actions](container-actions.md). diff --git a/docs/areas/scan-engine.md b/docs/areas/scan-engine.md new file mode 100644 index 0000000..7320191 --- /dev/null +++ b/docs/areas/scan-engine.md @@ -0,0 +1,215 @@ +# Scan engine + +## Scope + +When and how often the application refreshes its data in the background, how each refresh is recorded as a scan +run, how runs that never finished are repaired and how old scan data is removed. Operators rely on it through the +`DockerUpdateGuard:Scanning` interval and retention keys; the [Web UI](web-ui.md) relies on the scan-run records +for its scan history and status chips. What a single scan evaluates belongs to the feature areas listed under +*Not here*. + +## Behavior + +### Background jobs + +The host runs exactly seven recurring background jobs, all in the application process: + +| Job | What it triggers | Interval key (`DockerUpdateGuard:Scanning:…`) | Default | Allowed range | +| --- | --- | --- | --- | --- | +| Docker instance synchronization | Reconcile the configured Docker instances with the stored ones ([Docker instances](docker-instances.md)) | `DiscoveryIntervalMinutes` | 15 min | 1–1440 | +| Docker Hub account discovery | Refresh the images of the configured Docker Hub account ([Registries](registries.md)) | `DockerHubAccountDiscoveryIntervalMinutes` | 60 min | 1–10080 | +| Observed-image refresh | Scan every enabled observed image (tag metadata, base images, derived runtime) | dynamic, see below (`OwnImageBaseScanIntervalMinutes` is the floor) | 60 min | 1–10080 | +| Runtime container refresh | Scan the running containers of every enabled Docker instance | `RuntimeImageUpdateScanIntervalMinutes` | 30 min | 1–1440 | +| Resource statistics | Sample CPU, memory and network usage ([Docker instances](docker-instances.md)) | `ResourceStatisticsIntervalMinutes` | 5 min | 1–1440 | +| Vulnerability refresh | Re-assess the vulnerabilities of all live images ([Vulnerabilities](vulnerabilities.md)) | `VulnerabilityRefreshIntervalMinutes` | 180 min | 1–10080 | +| Scan cleanup | Repair unfinished runs and purge old data (see *Retention and cleanup*) | `CleanupIntervalMinutes` | 720 min | 1–10080 | + +- The jobs start after the startup sequence of [Hosting and configuration](hosting.md) has completed (database + reachable, migrations applied). +- Every job except scan cleanup runs once immediately when it starts and then waits one interval between runs. + Scan cleanup first performs its stale-run repair (see *Unfinished runs*) and runs its first full pass only after + one `CleanupIntervalMinutes` interval. +- The wait is measured from the end of one run to the start of the next, so runs of one job never overlap each + other and a long run delays the next one. +- The jobs are independent of each other: they are neither ordered nor coordinated, and two different jobs may run + at the same time. +- The interval is read again before every wait. A changed interval key takes effect from the next wait without a + restart; a wait already in progress is not shortened or extended. +- Each run works in its own unit of work (its own database context), so no state carries over between runs. + +### Dynamic observed-image interval + +The observed-image refresh stretches its interval with the number of enabled observed images so that scheduled +Docker Hub requests stay within a request budget: + +``` +budget = DockerHubRequestLimitPerWindow − DockerHubReservedManualRequestsPerWindow +interval = max(OwnImageBaseScanIntervalMinutes, ⌈DockerHubRequestLimitWindowHours × 60 × enabledImages / budget⌉) minutes +``` + +- With no enabled observed image the interval is `OwnImageBaseScanIntervalMinutes`. +- The count of enabled observed images is read again before every wait. +- Defaults and ranges: `DockerHubRequestLimitWindowHours` 6 (1–168), `DockerHubRequestLimitPerWindow` 200 + (1–100000), `DockerHubReservedManualRequestsPerWindow` 40 (0–100000). The reserved share must be smaller than + the limit per window, so the budget is always at least 1. +- With the defaults, up to 26 enabled images are scanned every 60 minutes; from 27 images on the interval grows by + about 2.25 minutes per image (e.g. 100 images → 225 minutes). +- The formula only spaces out scheduled runs. Manual scans are not counted against the budget, and the job does not + count the requests a run actually makes. Request parallelism and HTTP 429 handling belong to + [Registries](registries.md). + +### Scan runs + +Three kinds of work are recorded as scan runs; the other jobs (instance synchronization, account discovery, +resource statistics, cleanup) leave no scan-run record and report only through logs. + +| Type | One run per | Started by | +| --- | --- | --- | +| `ObservedImage` | observed image scanned | the observed-image refresh (one run per enabled image), or registering an image in the UI | +| `RuntimeContainer` | enabled Docker instance scanned | the runtime container refresh, or the manual update check in the UI | +| `Vulnerability` | vulnerability refresh | the vulnerability refresh (one run for all live images), or a manual refresh or per-image rescan in the UI | + +- A run records its type, status, trigger, start time, completion time, an optional error message and the observed + image or Docker instance it concerns. +- Trigger is `Scheduled` for runs started by a background job and `Manual` for runs started from the UI. The stored + values `Startup` and `Discovery` are reserved and not written today. +- A run is stored with status `Running` before any work begins and is updated once at the end to one of: + - `Succeeded` — every step completed; + - `Partial` — at least one step returned an incomplete result (for example an unsupported registry or one + container that could not be processed) while the run as a whole went on; + - `Failed` — a step reported a failure, or an unexpected error ended the run. +- The error message joins the distinct messages of all incomplete or failed steps, one per line; it is empty for a + `Succeeded` run. +- The stored values `NotSet` and `Pending` are reserved and not written today. +- A vulnerability refresh that is disabled by configuration, a refresh for an image version that does not exist and + an observed-image scan for an image that no longer exists create no run at all. +- A runtime container refresh first synchronizes the configured Docker instances; instances that are enabled in the + database but missing from or disabled in configuration are skipped without a run. +- After every recorded run the inventory gauges are refreshed and the run's duration and status are recorded as + metrics ([Telemetry](telemetry.md)). + +### Overlapping scans + +- Scans of the same observed image never run at the same time within one application process. A second request + for an image that is being scanned waits until the first scan has finished and then runs in full; it is not + skipped or merged. +- Scans of different observed images may run at the same time (for example a manual registration during the + scheduled refresh). +- Runtime container scans and vulnerability refreshes have no such guard: a manual run and a scheduled run of the + same kind may overlap. +- Within the observed-image refresh the images are scanned one after another; within the runtime container refresh + the instances are scanned one after another. Parallelism of vulnerability provider calls inside one refresh is + owned by [Vulnerabilities](vulnerabilities.md). + +### Failure isolation + +- A run of a background job that fails with an error is logged with the job name and its duration and does not stop + the job, the other jobs or the host; the job waits its interval and runs again. +- Inside a batch, an error while scanning one observed image or one Docker instance is logged and the batch goes on + with the next one. +- A failed run is not retried before the next scheduled run. `DockerUpdateGuard:Scanning:RetryCount` (default 2, + range 0–10) applies to individual outbound HTTP requests, not to runs ([Registries](registries.md)). +- Host shutdown cancels the run in progress; it is logged neither as completed nor as failed. + +### Unfinished runs + +A run can stay `Running` when the process stops or crashes during a scan. Such runs are repaired: + +- once when the cleanup job starts, and again at the beginning of every cleanup pass; +- a `Running` run counts as stale once it started longer ago than its threshold, which is twice the configured + interval of its job, but at least 24 hours: + - `ObservedImage`: 2 × `OwnImageBaseScanIntervalMinutes` (the configured floor, not the dynamic interval); + - `RuntimeContainer`: 2 × `RuntimeImageUpdateScanIntervalMinutes`; + - `Vulnerability`: 2 × `VulnerabilityRefreshIntervalMinutes`; +- a stale run is set to `Failed`, gets the repair time as completion time and the error message + "Marked as failed by cleanup: the run never completed (process restart or crash)". + +With the default intervals every threshold is 24 hours, so a run interrupted by a restart shows as `Running` for up +to a day. + +### Retention and cleanup + +Each cleanup pass computes a cutoff of now minus `DockerUpdateGuard:Scanning:RetainScanRunsDays` (default 30, range +1–3650) and removes, in one transaction: + +- **Scan runs:** only completed runs that no update finding, vulnerability finding, container snapshot or image + relationship refers to. Of these, a run is kept only while it is both among the 20 most recently completed and + not completed before the cutoff; all others are removed. Runs still referenced by data are kept regardless of age + and count, and `Running` runs are never removed. +- **Tag candidates:** those whose update finding was resolved before the cutoff, and those without a digest + regardless of age. +- **Update findings:** inactive findings resolved before the cutoff. +- **Vulnerability findings:** inactive findings of image versions that are no longer live, regardless of age, and + inactive findings of live image versions resolved before the cutoff. +- **Container snapshots:** snapshots recorded before the cutoff that have no active update finding and no recorded + container action. +- **Resource samples:** container and Docker instance samples recorded before the cutoff. + +After the removal the cleanup pass refreshes the inventory gauges ([Telemetry](telemetry.md)) and logs how many +tag candidates, update findings, vulnerability findings, snapshots and scan runs it removed. Nothing else (images, +observed images, Docker instances, relationships, tag selections) is removed by cleanup. + +### Live images + +An image version is *live* when it is: + +- the current version of any observed image (enabled or disabled), or +- running according to the most recent runtime container scan of any Docker instance (for an instance without a + runtime scan, the most recent snapshot of each of its containers counts), or +- a direct base image of one of the above. + +The vulnerability refresh assesses exactly the live image versions, and cleanup uses the same set to purge findings +of image versions that left the fleet. + +### Deployment assumption + +The engine assumes a single application instance per database. Its scan guard is local to one process and the jobs +are not coordinated through the database: several instances against one database duplicate the scheduled work and +may scan the same image at the same time. This is wasteful but does not corrupt the shared image catalog +([Image catalog](image-catalog.md)). + +## Limits and errors + +- Interval, budget and retention keys outside their ranges, or a reserved request share not smaller than the limit + per window, fail configuration validation with a message naming the key and the allowed range + ([Hosting and configuration](hosting.md)). +- The smallest possible interval of every job is one minute; there is no sub-minute schedule and no cron-style + schedule. +- A step that reports a failure or an unexpected error turns the run `Failed`; the error message of the run carries + the step's message as returned by the integration, or the error's own message. Integrations report expected + failures as results, not errors ([Registries](registries.md)). +- A run that cannot be saved at its end (for example because the database became unreachable) stays `Running` until + the stale-run repair marks it `Failed`. A vulnerability refresh retries this final save once on a fresh state + before giving up. +- If the database cannot be reached during a cleanup pass, the whole pass fails, nothing is removed, and the next + pass starts after one interval. + +### Known gaps + +- Reading a job's interval is outside the job's error handling. When it fails — a configuration reload with an + invalid value, or an unreachable database while the observed-image refresh counts the enabled images — the job + ends with an error and the host stops, instead of the job keeping its last known interval. + Tracked in [#281](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/281). +- The 20 most recently completed scan runs are a cap, not a minimum: an unreferenced run completed before the + `RetainScanRunsDays` cutoff is removed even when it is among the 20 most recent, so the scan history can be + empty after a long pause. Tracked in [#290](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/290). +- Runtime container scans and vulnerability refreshes have no per-subject guard: a manual and a scheduled run of the + same kind can run at the same time (see *Overlapping scans*). + Tracked in [#292](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/292). + +## Related decisions + +None yet. + +## Not here + +- What an observed-image scan resolves: tag metadata and image identity in [Image catalog](image-catalog.md) and + [Registries](registries.md), base images and derived runtimes in [Base images](base-images.md), recommended + updates in [Update detection](update-detection.md). +- Container discovery, snapshots and resource sampling in [Docker instances](docker-instances.md). +- Vulnerability providers, assessment status and the provider parallelism in + [Vulnerabilities](vulnerabilities.md). +- Docker Hub request parallelism, HTTP retry and the failure model of integrations in [Registries](registries.md). +- How scan runs and manual triggers are shown in the UI in [Web UI](web-ui.md). +- Metrics recorded per run and the inventory gauges in [Telemetry](telemetry.md). +- Configuration loading, validation and the startup sequence in [Hosting and configuration](hosting.md). diff --git a/docs/areas/telemetry.md b/docs/areas/telemetry.md new file mode 100644 index 0000000..c65bef8 --- /dev/null +++ b/docs/areas/telemetry.md @@ -0,0 +1,142 @@ +# Telemetry + +## Scope + +What the application emits about itself as logs, traces and metrics, how each signal is switched on and exported, +which application-specific spans and instruments exist and when they change, and what may never appear in them. +Operators rely on it to monitor the application in an OpenTelemetry backend; dashboards and alerts rely on the +names and attributes stated here. + +## Behavior + +### Signals and switches + +- There are three signals: logs, traces and metrics. Each is switched independently by `Telemetry:EnableLogging`, + `Telemetry:EnableTracing` and `Telemetry:EnableMetrics`; all three default to `true`. +- When all three are `false`, no OpenTelemetry pipeline is set up at all and the other `Telemetry` keys are not + validated. +- The platform's console logging is independent of these switches and always active. `Logging:LogLevel` filters apply + to console output and to exported logs alike (shipped defaults: `Information`, and `Warning` for + `Microsoft.AspNetCore`). +- The `Telemetry` section is read once at startup; changing it requires a restart. + +### Export + +- All enabled signals are exported over OTLP to one endpoint, `Telemetry:OtlpEndpoint`. It must be an absolute `http` + or `https` URI. No transport protocol is configured by the application, so the OTLP exporter's default (gRPC, for + example `http://collector:4317`) applies. +- Without `Telemetry:OtlpEndpoint`, enabled signals are collected in-process but nothing is exported. There is no + fallback exporter (no console or Prometheus exporter). +- An unreachable or failing collector affects only the export; scanning, the UI and the database are unaffected. + +### Resource + +Every exported signal carries the same resource: + +| Attribute | Value | +| --- | --- | +| `service.name` | `Telemetry:ServiceName` (default `DockerUpdateGuard`); must not be blank while any signal is enabled. | +| `service.version` | The environment variable `DockerUpdateGuard__DisplayVersion` (the container image sets it); otherwise the application's informational version, file version or assembly version, in that order; otherwise `unknown`. | +| `deployment.environment.name` | `Telemetry:Instance`, only when it is set. The shipped `appsettings.json` sets `Production`; there is no built-in default. | + +### Traces + +- Traces contain a span for every incoming HTTP request and every outgoing HTTP call, plus the application's own + spans from the source `DockerUpdateGuard`. +- Application spans emitted today, all of kind client: + + | Span name | Wraps | Attribute | + | --- | --- | --- | + | `dockerhub.request` | A Docker Hub repository lookup, tag lookup or base-image resolution | `docker.image.reference` (the full image reference) | + | `dockerengine.request` | Container discovery on one Docker instance | `docker.instance.name` (the configured instance name) | + +- The span names `scan.run`, `cve.request`, `portainer.request`, `portainer.action` and `persistence.operation` are + reserved for the application's source but not emitted (see *Known gaps*). + +### Logs + +- Exported log records include the category, the formatted message, active scopes and the structured message + parameters as individual attributes. +- Application log events have fixed event IDs and message templates; the startup and database events are + `1000`–`1006` and the inventory refresh is `1010`. Message parameters are named in PascalCase + (`{DockerInstanceName}`, `{ImageReference}`, `{StatusCode}`). + +### Metrics + +- Metrics contain the application's own instruments from the meter `DockerUpdateGuard`, plus the platform's + instruments for incoming HTTP requests, outgoing HTTP calls and the .NET runtime. +- Scan instruments, recorded once per finished scan run after its final status is known (scan runs are defined in + [Scan engine](scan-engine.md)): + + | Name | Type | Unit | Recorded | + | --- | --- | --- | --- | + | `dockerupdateguard.scans.total` | Counter | — | `+1` for every finished scan run | + | `dockerupdateguard.scans.failed` | Counter | — | `+1` for every scan run that finished with status `Failed` | + | `dockerupdateguard.scans.duration` | Histogram | `s` | The run's duration in seconds | + + All three carry `scan.type` (`ObservedImage`, `RuntimeContainer` or `Vulnerability`) and `result.status` (the + final status, for example `Succeeded`, `Partial` or `Failed`). `scans.failed` counts only runs that produced no + usable result; a run that ends `Partial` is not a failed run and is counted in `scans.total` only. +- Inventory gauges: + + | Name | Value | + | --- | --- | + | `dockerupdateguard.observed_images` | Number of observed images, enabled or not | + | `dockerupdateguard.runtime_containers` | Number of distinct containers (per Docker instance and container ID) that have a stored snapshot | + | `dockerupdateguard.base_images.deduplicated` | Number of distinct image versions used as a base image by at least one image | + | `dockerupdateguard.findings.update.active` | Number of active update findings | + | `dockerupdateguard.findings.cve.active` | Number of active vulnerability findings | + | `dockerupdateguard.findings.needs_review.active` | Number of active update findings that are tag recommendations | + +- Gauges report the values of their last refresh; observing them never queries the database. They are refreshed: + - once during startup, before requests are served (see [Hosting](hosting.md)); + - after every finished observed-image scan, runtime-container scan and vulnerability refresh; + - after every cleanup pass. +- Between refreshes a gauge can lag behind the database, for example after a manual registration that triggers no + scan. + +### Credentials and external data + +- Credential values never appear in log messages, log attributes or span attributes: not the Docker Hub PAT, the + Portainer password or API token, the bearer tokens obtained from Docker Hub, registries, Docker Scout or Portainer, + nor the database connection string. Credentials travel only in request headers, and headers are not recorded. +- Database statements are logged without parameter values. +- What logs and spans may carry about external systems: configured instance names, image references, configured base + URLs, the Docker Hub account name, HTTP methods and status codes, counts, durations and error messages from + external tools. Retry log lines carry the request URL without its query string. + +## Limits and errors + +- `Telemetry:OtlpEndpoint` set but not an absolute `http`/`https` URI: startup fails with + `'Telemetry:OtlpEndpoint' must be an absolute http or https URI`. +- `Telemetry:ServiceName` blank while any signal is enabled: startup fails with + `'Telemetry:ServiceName' must be configured when telemetry is enabled`. +- Both checks run while services are registered, before the host is built; a failure aborts the process before the + database is contacted. +- Export failures are handled inside the exporter and are never surfaced to users or scans. + +### Known gaps + +- Scans, vulnerability provider calls, Portainer calls and persistence have no application span; the reserved span + names `scan.run`, `cve.request`, `portainer.request`, `portainer.action` and `persistence.operation` are never + emitted. Tracked in [#83](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/83). +- `dockerupdateguard.runtime_containers` counts every container that still has a stored snapshot, including + containers that have since stopped or been removed, until cleanup deletes their snapshots; it does not count only + the currently known containers. Tracked in [#294](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/294). +- Configured base URLs (Docker instance, Portainer) are logged verbatim, and retry log lines keep the user-info part + of a URL, so credentials embedded in a URL (`https://user:secret@host`) reach the logs. + Tracked in [#295](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/295). + +## Related decisions + +None yet. + +## Not here + +- Deploying the collector and the Grafana/LGTM stack: `INSTALL.md` and `docs/docker-compose.full.yml`. +- What a scan run is and when scans run: [Scan engine](scan-engine.md). +- What counts as a finding, a tag recommendation or a base image: [Update detection](update-detection.md), + [Vulnerabilities](vulnerabilities.md), [Base images](base-images.md). +- Secrets in the UI: [Web UI](web-ui.md); secrets in configuration: [Hosting](hosting.md). +- Runtime warnings for `SkipCertificateValidation` and `AllowInsecureHttp`: [Docker instances](docker-instances.md) + and [Container actions](container-actions.md). diff --git a/docs/areas/update-detection.md b/docs/areas/update-detection.md new file mode 100644 index 0000000..7287a4a --- /dev/null +++ b/docs/areas/update-detection.md @@ -0,0 +1,195 @@ +# Update detection + +## Scope + +Which update DockerUpdateGuard recommends for a running container image, given the tags its registry publishes: how +tags are parsed and compared, how floating alias tags are resolved to a concrete version, how conservative a major +upgrade is, which assessment status a runtime container gets, and how update findings with their ranked tag candidates +and the manual tag selection per container are kept. Users rely on it to decide what to update; the UI and the +inventory metrics rely on the stored assessments and findings. + +## Behavior + +### Tag formats + +- A **semantic version tag** has the form `[v|V]..[-]` with decimal components, e.g. + `1.27.3`, `v2.0.0-alpine`, `8.0.11-bookworm-slim`. A component too large for a 32-bit integer makes the tag + non-semantic. +- A **version line tag** has the form `[v|V].[-]`, e.g. `8.0` or `1.27-alpine`. It is treated as + an alias of the newest patch of that line, never as a version itself. +- A **year-prefixed tag** has the form `-`, e.g. `2022-CU12-ubuntu-22.04`. +- Every other tag (`latest`, `stable`, `alpine`, ...) is an **alias tag**. +- Tag names are compared case-insensitively throughout. When the registry reports the same tag name twice, only the + most recently published entry counts. + +### Variant families and ordering + +- The suffix of a semantic version tag is split at `-` into segments. A segment whose leading letters are one of + `alpha`, `beta`, `rc`, `pre`, `preview`, `dev`, `snapshot`, `nightly`, `canary`, `milestone`, `ea` (any case) is a + **pre-release segment**; all others are **variant segments**. +- The **variant family** of a semantic tag is its variant segments, each reduced to its leading letters (or kept + whole when it has none), joined by `-`: `1.2.3-alpine3.19` and `1.2.4-alpine3.20` are both family `alpine`; + `1.2.3-rc1` and `1.2.3` are both the plain family. +- Two semantic tags are comparable only within the same variant family. A tag of another family is never a candidate, + so an Alpine tag is never recommended for a Debian one. +- Comparable tags order by major, minor, patch; then by their variant segments in natural order (digit runs compared + numerically, text case-insensitively); then a release ranks above any pre-release of the same version; then + pre-release segments in natural order (`rc2` below `rc10`). +- Year-prefixed tags are comparable only within the same year and the same year-variant family. The year-variant + family is the suffix after its leading servicing segments (segments that start with letters and contain a digit, + such as `CU12`), lower-cased; it is empty when that rest contains no letter. Within a family, tags order by the + natural order of the suffix. + +### Alias resolution + +- An alias or version line tag is resolved to the semantic version tag that carries the same digest. The digest is the + running digest, or, when that is unknown, the registry digest of the alias tag. Without any digest nothing is + resolved. +- For a version line tag only semantic tags of the same major, minor and variant family qualify. +- When several tags qualify, the order of preference is: the variant family of a given reference tag (used when + resolving a recommended alias against the container's current version), then plain tags without a variant, then the + most recently published, then the highest version. +- The resolved tag is stored on each container snapshot as the container's *resolved version*. When it cannot be + resolved in a scan, the resolved version of the most recent earlier snapshot of the same container and the same + image version is carried over, provided the running digest is known. +- When an update is available, the snapshot also stores the *available update version*: the recommended tag itself + when it is a semantic or year-prefixed tag, otherwise the semantic tag the recommended alias resolves to. + +### Input of an evaluation + +- A runtime container is evaluated against at most 250 repository tags requested from its registry for the image's + operating system and architecture ([Registries](registries.md)). +- Before evaluation, tags are dropped that were published before the current tag or that order below the current + version (or, for `latest`, below the version `latest` resolves to), unless they carry the running digest. The + current tag itself is always kept, with the metadata of the direct lookup of the current tag when that succeeded. + +### Evaluation rules + +An evaluation returns one of *up to date*, *update available*, *needs review* or *unknown*, a summary, optional +details, an optional recommended tag and digest, and at most **50** candidates. The first rule that applies wins: + +1. **No tags** — *unknown*, "No registry tag information is available". +2. **`latest` unchanged** — the current tag is `latest` and the running digest equals the registry digest of `latest`: + *up to date*. +3. **Semantic current tag** — candidates are the comparable tags that order above the current tag, are not + pre-releases (unless the current tag is one), do not carry the running digest and were published after the + current tag (when both timestamps are known). They are ordered highest first, ties broken by newest publication, + and filtered by the major-upgrade policy below. If candidates remain, the first is recommended (*update + available*). Otherwise, if the current tag was republished with a different digest, that is the update; otherwise + *up to date*. +4. **Year-prefixed current tag** — candidates are the tags of the same year and year-variant family that order above + the current tag, with the same digest and publication conditions; highest first. No major-upgrade policy applies, + and no other year is ever recommended. Without candidates, a republished current tag is the update; otherwise *up + to date*. +5. **Republished alias** — the registry digest of the current tag differs from the running digest: *update available* + recommending the current tag with its new digest. Its candidates are the tags carrying that new digest: the + current tag first, then tags of the same version line, then semantic tags, then newest and highest. +6. **Version line alias** — the running digest matches a semantic tag of the line: that tag is the current version and + rule 3 is applied to it. The resolved current version is added to the candidates. Without a successor the result is + *up to date*, "The running digest matches version tag '…'". +7. **Anything else** — every other tag that has a digest and was published after the current tag (when both + timestamps are known; newest first, at most 50) is a candidate for *needs review*, "Alternative tags are + available and require manual review". With no such tag the result is *up to date*. + +### Major-upgrade policy + +- When a successor exists in the current major line, the highest of them is recommended. Higher majors are kept as + further candidates but are never recommended. +- When the only successors are in higher majors, only the **next** major (the lowest one above the current) can be + recommended, and only when all of the following hold: + - the next major has at least `DockerUpdateGuard:Scanning:MajorUpgradeMinimumReleaseCount` distinct candidate + versions (default `2`, range `0`–`100`, `0` disables this condition); + - and, unless `DockerUpdateGuard:Scanning:MajorUpgradeMinimumAgeDays` is `0` (default `30`, range `0`–`3650`): + - the earliest known release of the next major (same variant family, with a publication timestamp) is exactly + `.0.0`, so a truncated tag history defers the upgrade; + - that first release was published at least the configured number of days ago; + - the most recent release of the current major line is known, at least that many days old, and not later than the + first release of the next major (the current line is dormant, not maintained next to the new one). +- A recommended major upgrade carries the details "No newer version is available in the `.x` version line". +- A deferred major upgrade yields *up to date*, "No newer version was found in the `.x` version line", with the + details "Tag '…' introduces a new major version and is not recommended yet". A republished current tag (rule 3) + takes precedence over this result. +- The age conditions are measured against the wall clock at evaluation time; a deferred upgrade becomes a + recommendation in a later scan without any other change. + +### Runtime container assessment + +Each container snapshot ([Docker instances](docker-instances.md)) carries an update assessment status and a message +(the summary, followed by the details when present): + +| Status | When | +| --- | --- | +| Not evaluated | Initial value of a new snapshot | +| Up to date | Evaluation result *up to date* | +| Update available | Evaluation result *update available* | +| Manual review required | Evaluation result *needs review* | +| No tag data | Evaluation result *unknown*, or the registry reports the repository as not found | +| Unsupported | No registry client supports the image's registry | +| Failed | The tag lookup failed for any other reason, or processing the container raised an error; the message is the failure message | + +A container that ends *Failed*, *Unsupported* or with *No tag data* because of the registry answer makes its runtime +scan run *partial*, with the message added to the run ([Scan engine](scan-engine.md)). + +### Update findings and tag candidates + +- For *update available* a finding of type *runtime image update* is created, for *needs review* one of type *tag + recommendation*. Each finding references the scan run, the container snapshot, the image version it is about and, + when a tag is recommended, the catalog image version of the recommended tag and digest + ([Image catalog](image-catalog.md)). +- The candidates of the evaluation are stored as tag candidates with tag, digest, publication time, the evaluation + summary as reason, a rank equal to the candidate's 1-based position in the evaluation and a *recommended* flag on the + candidate whose tag equals the recommended tag. Candidates without a digest are not stored, since they offer no + reliable update target; a finding can therefore hold fewer candidates than the evaluation produced, and ranks may + have gaps. + A finding holds each tag/digest pair at most once. +- Whenever a container was evaluated (including *no tag data* and *unsupported*), all active findings of the same + container on the same Docker instance are first deactivated: they become inactive, get a resolution timestamp and + lose their tag candidates. When the tag lookup failed, the previous findings stay active. +- After a runtime scan of an instance, the active findings of containers that were no longer discovered are + deactivated the same way. +- Findings of type *base image update* exist in the data model but are not produced. Findings of type *derived base + runtime update* are owned by [Base images](base-images.md). + +### Manual tag selection + +- A user can select one tag, with an optional digest, per runtime container (Docker instance and container id), and + clear it again. A new selection replaces the previous one; tag and digest are stored trimmed, an empty digest as + none, together with the registry repository of the container's latest snapshot and the selection time. +- Clearing a container without a selection does nothing. +- The selection is a record of the user's choice only: it does not change the evaluation, the findings or any + container action. It is shown while the container still runs an image of the same repository it was made for. +- A selection is removed together with its Docker instance or registry repository. + +## Limits and errors + +- At most 250 tags per runtime container evaluation and at most 50 candidates per finding. +- Stored lengths: tag candidate tag 200 characters, digest 255, reason 500; finding summary 500, details 2000; + resolved and available update version 200; assessment message 2000. +- Selecting a tag for a container that has no snapshot fails with "Runtime container '/' + was not found"; nothing is stored. +- Evaluation itself performs no I/O and has no failure state of its own; registry failures reach it only as missing + tags and are reported through the assessment status above, with the registry's failure message. + +### Known gaps + +- The manual tag selection is informational only: it does not pin the recommendation or influence any container + action. Tracked in [#291](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/291). +- Findings of type *base image update* are defined in the data model but never produced. Tracked in + [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). +- The major-upgrade age conditions read the system clock directly instead of the application's time source, so they + cannot be controlled independently of the real time. + Tracked in [#289](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/289). + +## Related decisions + +None yet. + +## Not here + +- Fetching tags, digests and publication times, registry support and retries: [Registries](registries.md). +- Parsing and normalizing image references, the image version catalog: [Image catalog](image-catalog.md). +- Base-image resolution, derived .NET/nginx runtime findings and observed-image findings: [Base images](base-images.md). +- When runtime scans run, scan runs, retention and cleanup of findings and candidates: [Scan engine](scan-engine.md). +- Container discovery and snapshots: [Docker instances](docker-instances.md). +- Applying an update through Portainer: [Container actions](container-actions.md). +- How findings, candidates and the selection are displayed: [Web UI](web-ui.md). diff --git a/docs/areas/vulnerabilities.md b/docs/areas/vulnerabilities.md new file mode 100644 index 0000000..2fe9932 --- /dev/null +++ b/docs/areas/vulnerabilities.md @@ -0,0 +1,181 @@ +# Vulnerabilities + +## Scope + +How DockerUpdateGuard enriches image versions with known vulnerabilities: the choice of provider (Trivy, Docker Scout +or none), how each provider is called and its answer mapped, which image versions a refresh covers, the vulnerability +assessment per image version, the lifecycle of vulnerability findings, the manual rescan and the configuration hints. +Users rely on it to see which running or observed images carry which CVEs; operators rely on it to configure a +provider; the UI and the inventory metrics rely on the stored assessments and findings. + +## Behavior + +### Configuration + +All keys live under `DockerUpdateGuard:Vulnerabilities`: + +| Key | Default | Rule | +| --- | --- | --- | +| `Enabled` | `false` | Switches vulnerability refresh on | +| `Provider` | `None` | `None`, `DockerScout` or `Trivy`; must not be `None` when `Enabled` is `true` | +| `TrivyBaseUrl` | none | Absolute `http`/`https` URI; required when `Enabled` is `true` and `Provider` is `Trivy` | +| `TrivyExecutablePath` | `trivy` | Executable to start; resolved through `PATH`; an empty value means `trivy` | +| `DockerScoutLoginUrl` | `https://hub.docker.com/v2/users/login` | Absolute `http`/`https` URI, always validated | +| `DockerScoutBaseUrl` | `https://api.scout.docker.com` | Absolute `http`/`https` URI, always validated | +| `RequestTimeoutSeconds` | `120` | `1`–`300` | +| `MaxParallelScans` | `1` | `1`–`8` | + +- A configuration that breaks one of these rules fails startup ([Hosting](hosting.md)). +- The provider is chosen anew at the start of every refresh from the current configuration, so a reloaded + configuration takes effect with the next refresh. When `Enabled` is `false`, a refresh does nothing: no provider is + called, no scan run is recorded and stored assessments and findings stay as they are. +- Docker Scout uses the Docker Hub credentials `DockerUpdateGuard:DockerHub:UserName` and + `DockerUpdateGuard:DockerHub:Pat`; they are not required by validation. + +### What a refresh covers + +- A full refresh scans every **live** image version: the current image version of every observed image, the image + versions of the most recent runtime container scan of each Docker instance (for an instance without such a scan, + the latest snapshot of each container), and the recorded base images of all of these + ([Base images](base-images.md)). +- A full refresh runs on the schedule `DockerUpdateGuard:Scanning:VulnerabilityRefreshIntervalMinutes` (default `180`, + range `1`–`10080`, see [Scan engine](scan-engine.md)) and can be started manually from the scan history page. +- A **manual rescan** of a single image version can be started from the observed image, own image and runtime + container lists. It follows the same rules for that one image version. An image version that does not exist is + ignored without a scan run. +- Every refresh is recorded as one scan run of type *vulnerability* with trigger *scheduled* or *manual*, also when it + covers no image version. +- Provider calls run in batches of `MaxParallelScans` image versions at a time; the results of a batch are applied and + saved one image version after the other. + +### Assessment per image version + +Every scanned image version gets an assessment status, the provider it came from, a message, the check time and the +scan run: + +| Status | When | Message | +| --- | --- | --- | +| Not scanned | No assessment stored yet | — | +| No findings | Provider succeeded with no advisory | "The vulnerability provider reported no findings" | +| Findings detected | Provider succeeded with advisories | "The vulnerability provider reported N finding(s)", N after deduplication | +| Not configured | Provider not configured (missing Trivy URL or Docker Hub credentials) | Provider message | +| Unsupported | Provider cannot scan this image (Docker Scout and a registry other than `docker.io`) | Provider message | +| Unavailable | Image or tag not found by the provider | Provider message | +| Failed | Any other provider failure | Provider message | + +- A failed assessment leaves the image version's findings untouched. +- A failed assessment adds its message to the scan run. The run ends *failed* once any provider call returned + *failed* (or the run itself failed), otherwise *partial* when any assessment was not successful, otherwise + *succeeded*. + +### Findings + +- A finding is identified within an image version by advisory id and affected package, compared case-insensitively. + When a provider reports the same pair more than once, the entry with the highest severity, then the highest CVSS + score, is kept. +- For each reported advisory: a matching active finding is updated; a matching inactive finding (the most recently + resolved one) is reactivated with a new detection time and the current scan run, and updated; otherwise a new + finding is created. Updating overwrites title, severity, source, installed and fixed version, fix status, package + class and type, CVSS score, summary and reference URL. +- A finding that was active before a successful scan of its image version and is not reported again is deactivated + with a resolution time and the resolving scan run. +- At the end of a full refresh, every active finding of an image version that is neither live nor scanned in this run + is deactivated the same way. A manual rescan never does this. +- At most one active finding exists per image version, advisory id and affected package. +- Severity is one of *not set*, *low*, *medium*, *high*, *critical*. The source is the configured provider (*Trivy* or + *Docker Scout*). Fix status is one of *not set*, *fixed*, *affected*, *will not fix*, *fix deferred*, *end of life*. + +### Trivy + +- Trivy runs as a local process in client mode against the Trivy server at `TrivyBaseUrl`; the image is analysed by + the process, only the vulnerability database lookup goes to the server. No shell is involved; each argument is + passed separately: + `image --server --format json --quiet --scanners vuln --pkg-types os,library + --timeout s `. +- The target is `:` for `docker.io` images and `/:` otherwise, followed by + `@` when the digest is known. +- The process gets `RequestTimeoutSeconds` plus 10 seconds. When that passes, the process and its children are killed + and the result is *failed*, "Trivy scan timed out after N seconds". +- A non-zero exit code whose error output contains `MANIFEST_UNKNOWN`, `manifest unknown`, or both `manifest` and + `not found` (any case) means the image is gone from the registry: *unavailable*, "The image is no longer available + in the registry. The tag was likely re-published with a newer digest, leaving this version orphaned." Any other + non-zero exit code is *failed*, "Trivy exited with code X: ". +- The JSON report on standard output is mapped per vulnerability of every result block: advisory id from + `VulnerabilityID`; title from `Title`, else the advisory id; severity from `Severity` (`LOW`, `MEDIUM`, `HIGH`, + `CRITICAL`, any case, everything else *not set*); affected package, installed and fixed version from `PkgName`, + `InstalledVersion`, `FixedVersion`; fix status from `Status` (`fixed`, `affected`, `will_not_fix`, `fix_deferred`, + `end_of_life`, any case, everything else *not set*); package class and type from the result block's `Class` and + `Type`; summary from `Description`; reference URL from `PrimaryURL`, else the first of `References`. +- The CVSS score is the NVD v3 score, else the first v3 score of any source, else the NVD v2 score, else the first v2 + score of any source, else none. + +### Docker Scout + +- Docker Scout scans only `docker.io` images, by tag; the digest is not used. A repository without a namespace is + looked up under `library`. +- The provider logs in by posting the Docker Hub user name and token to `DockerScoutLoginUrl` and caches the returned + token for the whole process until one minute before the expiry stated in the token, or for five minutes when the + expiry cannot be read. Only one login runs at a time. A failed login gives *failed*, "Docker Hub authentication + failed". +- Vulnerabilities are read from `/v1/repositories///tags//vulnerabilities`, + following the `next` link of each page, for at most **50 pages**. When the cap is reached, the advisories read so + far are returned as a successful result. +- HTTP 401 discards the cached token, logs in once more and reads again from the first page; a second 401 gives + *failed*, "Docker Scout rejected the authentication token". HTTP 404 gives *unavailable*, "Image not + found in Docker Scout". Any other unsuccessful status gives *failed*, "Docker Scout request failed with status X". +- Each HTTP request is limited to `RequestTimeoutSeconds`. +- Mapping: advisory id and title from `id`, severity from `severity` (as for Trivy), summary from `description`, + reference URL from the first of `links`. Docker Scout supplies no package, versions, fix status or CVSS score. + +### Configuration hints + +The dashboard shows at most one hint about the vulnerability configuration: + +- *Scanning disabled* when `Enabled` is `false`. +- *Trivy base URL missing* when `Provider` is `Trivy` and the most recent completed vulnerability scan run (by start + time) ended *partial* or *failed* with a message containing "TrivyBaseUrl is not configured". +- *Docker Hub credentials missing* when `Provider` is `DockerScout` and that run ended *partial* or *failed* with a + message containing "Docker Hub credentials are required for Docker Scout". +- No hint otherwise. + +## Limits and errors + +- Providers never raise errors for expected failures; they report *not configured*, *unsupported*, *not found* or + *failed* with a message ([Registries](registries.md)). An unexpected error inside a provider call becomes *failed* + with "Trivy scan failed: …" or "Docker Scout scan failed: …". Cancellation is not converted. +- An unexpected error during a refresh ends the scan run *failed* with the error message; work saved before stays + saved. If saving the final run state fails, it is retried once. +- Stored lengths: advisory id 100 characters, title 300, affected package 200, installed and fixed version 100 each, + package class and type 50 each, CVSS score 5 digits with 2 decimals, assessment message 2000; summary and reference + URL are unlimited. +- Messages are built from the image reference and the provider's error text; no credentials or tokens are put into + them. + +### Known gaps + +- Docker Scout follows the `next` link of a response to whatever host it names and sends the bearer token along; the + same gap in the registry clients is owned by [Registries](registries.md). Tracked in + [#264](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/264). +- Docker Scout requests do not go through the transient-failure retry the other outbound HTTP calls use, and Docker + Scout scans the tag, not the digest, so an image version whose tag was republished is assessed with the advisories + of the newer image. Tracked in [#267](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/267). +- Provider values longer than the stored lengths (for example a Trivy title over 300 characters) are not shortened; + saving them fails and ends the whole refresh *failed*, leaving the remaining image versions unscanned. Tracked in + [#271](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/271). +- Some values are kept although nothing produces them: validation makes the *Trivy base URL missing* hint and the + Trivy *not configured* result unreachable (unless an invalid reloaded configuration is still served), and the + finding sources *registry advisory* and *manual* exist in the data model but are never produced. Tracked in + [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). + +## Related decisions + +None yet. + +## Not here + +- Refresh schedule, scan run statuses, stuck-run repair and cleanup of resolved findings: [Scan engine](scan-engine.md). +- Which base images an image version has: [Base images](base-images.md). +- Docker Hub credentials, registry access and the retry of transient HTTP failures: [Registries](registries.md). +- Validation at startup, the bundled Trivy CLI in the container image: [Hosting](hosting.md). +- How assessments, findings, severities and the rescan control are displayed: [Web UI](web-ui.md). +- Inventory gauges for active findings: [Telemetry](telemetry.md). diff --git a/docs/areas/web-ui.md b/docs/areas/web-ui.md new file mode 100644 index 0000000..c95b674 --- /dev/null +++ b/docs/areas/web-ui.md @@ -0,0 +1,164 @@ +# Web UI + +## Scope + +The browser interface of DockerUpdateGuard: its pages and routes, what each page shows, which actions a user can +start from it, how open pages pick up changes, the error and not-found pages, and light/dark mode. Users rely on it +to read the state the other areas compute and to start manual scans; it computes no findings of its own. + +## Behavior + +### General + +- The UI is a server-rendered web application. Every page is rendered on the server first and then stays + interactive over a persistent connection to the server; there is no client-side application logic beyond that + connection. Data loaded for the first render is reused when the page becomes interactive, not loaded twice. +- There is no sign-in. Everyone who can reach the application can see every page and start every action + ([Hosting and configuration](hosting.md) owns the security posture). +- All pages share one layout: a navigation drawer, a top bar and the page content. The drawer collapses into a menu + button on narrow screens. +- The top bar shows the product name, the dark-mode toggle, the number of *protected assets* (manual observed images + plus own images plus runtime containers) and *last activity* (the start time of the most recent scan run, or "No + scans yet"). The top bar reloads these values on every navigation and on every dashboard refresh signal (see + *Refresh without reload*). +- The navigation lists Dashboard, My Images, Observed Images, Runtime Containers, Docker Instances, Base Images, + Vulnerabilities and Scan History. *My Images* is listed only when both `DockerUpdateGuard:DockerHub:UserName` and + `DockerUpdateGuard:DockerHub:Pat` are set; *Base Images* only when at least one base image has been resolved. Both + conditions are evaluated when the session starts. The pages stay reachable by their routes either way. +- The drawer shows the application version: `DockerUpdateGuard:DisplayVersion` when set, otherwise the version of the + running build, otherwise "unknown". +- Timestamps are shown as short date and short time in the time zone and culture of the server process, not of the + browser. +- While a page loads its data it shows placeholder skeletons; a list without data shows an explanatory empty state. + +### Pages and routes + +| Route | Page | Shows | +| --- | --- | --- | +| `/` | Dashboard | Counts of my images, manual observed images, Docker instances, runtime containers and resolved base images; active update findings; active derived-runtime warnings of own images; active vulnerability findings of currently running containers with a per-severity breakdown; a configuration hint for vulnerability scanning; the 10 most recent scan runs | +| `/my-images` | My Images | Images discovered from the configured Docker Hub account, with latest scan status and message, number of running containers, derived-runtime hint and vulnerability status | +| `/my-images/{id}` | My Image details | One observed image: reference, latest scan, derived-runtime hint, running containers, base image chain, update findings with tag candidates, and — when there are any — vulnerability findings and its 20 most recent scan runs | +| `/observed-images` | Observed Images | The registration form and the manually registered images with latest scan, active update findings, derived-runtime hint and vulnerability status | +| `/observed-images/{id}` | Observed Image detail | One observed image: reference, latest scan, whether it is an own or a manual image, derived-runtime hint, running containers (own images only), base image chain, vulnerability findings and update findings with tag candidates; no scan history | +| `/runtime-containers` | Runtime Containers | Every currently known container with instance, tag (and the version an alias resolves to), CPU, memory and network usage with a short trend of up to the 12 latest samples, update state, available update and vulnerability status; a link to the own image when the container runs one | +| `/runtime-containers/{instanceId}/{containerId}` | Runtime Container detail | One container: image and digest, Compose project, linked own image, update state and message, derived-runtime hint, base image chain, resource usage history, manual tag preference, available tag candidates, update findings, vulnerability findings and the scans of its 20 most recent snapshots | +| `/docker-instances` | Docker Instances | Configured Docker instances with endpoint, connection kind, latest runtime scan status and time, current resource usage and number of containers; with exactly one instance its current usage and usage history are shown on this page directly | +| `/docker-instances/{id}` | Docker Instance detail | The containers currently observed on one instance with linked own image and resource usage | +| `/base-images` | Base Images | Resolved base images with the images built on them, the runtime containers and number of observed images that use them, and their vulnerability status | +| `/vulnerabilities` | Vulnerabilities | Active advisories across all images, one row per advisory, with severity and CVSS score, affected packages, fixed versions, number of affected images and running containers; a row expands to the affected images | +| `/scan-history` | Scan History | The 20 most recent scan runs with type, status, trigger, target, start time and error message | +| `/error` | Error | "Unexpected error" — the request could not be completed | +| `/not-found` | Not found | "Page not found" with a link back to the dashboard | + +- Detail routes accept only identifiers in GUID format (the container identifier of the runtime container route is + free text); any other path renders the not-found page. +- Resource usage histories on detail pages show up to the 48 most recent samples of the last 24 hours. +- The Vulnerabilities page filters by free text (advisory identifier, title or package, case-insensitive) and by + severity (All, Critical, High, Medium, Low), shows 25 rows per page with page sizes 25, 50 and 100, and links an + advisory to its reference only when the reference is an absolute `http` or `https` address; the link opens in a + new tab. +- Which values the pages show and how they are computed (update state, recommended tag, vulnerability status, + derived runtime, base image chain, resource values) is owned by the feature areas listed under *Not here*. + +### Actions + +| Action | Where | Effect | +| --- | --- | --- | +| Register and scan | Observed Images | Registers an image from name, image reference and optional description ([Image catalog](image-catalog.md)), scans it immediately as a manual observed-image scan and opens its detail page | +| Check image updates | Runtime Containers | Runs a manual runtime container scan of all enabled Docker instances | +| Refresh vulnerabilities | Scan History, shown only when vulnerability scanning is enabled | Runs a manual vulnerability refresh of all live images; the page also shows when the next scheduled refresh is expected | +| Rescan (per image) | Observed Images, My Images, Runtime Containers | Runs a manual vulnerability refresh of that one image version; next to it the time of the last check or "Never scanned" | +| Select / Clear selection | Runtime Container detail | Stores or removes a manual tag preference for that container ([Update detection](update-detection.md)); it does not change or redeploy the container | + +- Every action runs while the user waits: its button is disabled and shows progress until the action has finished, + and the page then reloads its data. Starting the same action twice from one page is not possible while it runs. +- Manual scans are recorded as scan runs with trigger `Manual` ([Scan engine](scan-engine.md)). +- The per-image rescan is offered even when vulnerability scanning is disabled; it then completes without doing + anything and records no run. +- Name and image reference are required for a registration; the form reports a missing value before anything is + sent. +- The expected time of the next scheduled vulnerability refresh is an estimate: the start of the most recent + vulnerability run of any trigger plus `VulnerabilityRefreshIntervalMinutes`, or "shortly (no run recorded yet)". +- The UI offers no container actions (restart, redeploy, update) and no way to edit configuration, delete images or + enable and disable them. + +### Refresh without reload + +- After a manual scan, registration or rescan finishes, every open dashboard and every open top bar in every browser + session connected to the same application process reloads its data without a page reload. +- Scheduled background runs do not send this signal: their results become visible when a page is opened or + navigated to again (the top bar also refreshes on every navigation). No page polls the server. + +### Vulnerability configuration hint + +The dashboard shows at most one hint about vulnerability scanning: + +- scanning disabled: how to enable it (`DockerUpdateGuard:Vulnerabilities:Enabled` and a provider); +- Trivy not configured: set `DockerUpdateGuard:Vulnerabilities:TrivyBaseUrl`; +- Docker Scout not configured: set `DockerUpdateGuard:DockerHub:UserName` and `DockerUpdateGuard:DockerHub:Pat`. + +When the hint applies is owned by [Vulnerabilities](vulnerabilities.md). The hint names configuration keys only, +never their values. + +### Light and dark mode + +- The UI has a light and a dark theme, switched with the toggle in the top bar. +- The choice is remembered per browser in encrypted browser storage. Without a stored choice the theme follows the + operating system's color-scheme preference, determined after the first render; the server-rendered first paint + is always light. +- If the browser storage cannot be read or written, the toggle still works for the session and the choice is not + remembered. +- Users who ask their system for reduced motion get (practically) no transitions or animations. + +### Secrets + +- The UI never shows configured secrets: the Docker Hub token, Portainer credentials and token and the database + connection string are neither displayed nor sent to the browser. Pages show only whether an integration is + configured, and the Docker Hub user name as the title of the My Images list. +- Docker instance endpoints are shown as configured (see *Known gaps*). + +## Limits and errors + +- An unhandled error during a request outside the Development environment renders the `/error` page; an unknown + route or an HTTP error status renders the `/not-found` page. The HTTP-level handling is owned by + [Hosting and configuration](hosting.md). +- An error in a user action is shown as an error message on the page that started it (for the per-image rescan, next + to its button); the page stays usable and the action can be started again. The message is the error's own message. +- Error messages of scan runs are shown as stored ([Scan engine](scan-engine.md)) in the dashboard, the scan history, + the latest-scan columns and the detail pages. +- If the connection to the server is lost, the page stops being interactive until the browser reconnects or the page + is reloaded. +- Lists are not paged except the Vulnerabilities page; all other lists show every item or the fixed number stated + above. + +### Known gaps + +- The scan status chips (dashboard, scan history, observed image and My Images lists) color only `Failed`, `Running` + and `Pending`; `Succeeded` and `Partial` are shown without color, so a partial run is not marked as a warning. + Tracked in [#274](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/274). +- A detail route with a well-formed identifier that does not exist (for example a deleted observed image) keeps + showing the loading skeleton instead of a not-found state. + Tracked in [#40](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/40). +- Docker instance endpoints are not masked: an endpoint that embeds credentials as user information in its URL is + shown with them. Tracked in [#295](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/295). +- The Scan History and dashboard descriptions mention discovery and cleanup runs, although those jobs record no scan + runs ([Scan engine](scan-engine.md)). + Tracked in [#278](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/278). + +## Related decisions + +None yet. + +## Not here + +- How update states, recommended tags, tag candidates and manual tag preferences are evaluated in + [Update detection](update-detection.md). +- Image registration rules and reference formats in [Image catalog](image-catalog.md). +- Base image chains, shared base images and derived runtime hints in [Base images](base-images.md). +- Vulnerability status, severities, fix status and the configuration hint rules in + [Vulnerabilities](vulnerabilities.md). +- Container discovery and resource values in [Docker instances](docker-instances.md). +- Scan runs, schedules and retention in [Scan engine](scan-engine.md). +- Container actions through Portainer in [Container actions](container-actions.md). +- Error and status-code middleware, HTTPS and the missing authentication in + [Hosting and configuration](hosting.md). diff --git a/docs/decisions/0001-quality-gates-before-the-pull-request.md b/docs/decisions/0001-quality-gates-before-the-pull-request.md index 006e0b9..942b0e6 100644 --- a/docs/decisions/0001-quality-gates-before-the-pull-request.md +++ b/docs/decisions/0001-quality-gates-before-the-pull-request.md @@ -2,6 +2,7 @@ - **Status:** Accepted - **Date:** 2026-10-03 +- **Area:** — - **Source:** Squad adopted from Squad-Spec-Repository-Template - **Supersedes:** — diff --git a/docs/decisions/0002-squash-merge-pull-requests.md b/docs/decisions/0002-squash-merge-pull-requests.md index 972befc..2df5731 100644 --- a/docs/decisions/0002-squash-merge-pull-requests.md +++ b/docs/decisions/0002-squash-merge-pull-requests.md @@ -2,6 +2,7 @@ - **Status:** Accepted - **Date:** 2026-10-03 +- **Area:** — - **Source:** Squad adopted from Squad-Spec-Repository-Template - **Supersedes:** — diff --git a/docs/decisions/0003-squad-working-records-off-main.md b/docs/decisions/0003-squad-working-records-off-main.md index ba57acc..539bd26 100644 --- a/docs/decisions/0003-squad-working-records-off-main.md +++ b/docs/decisions/0003-squad-working-records-off-main.md @@ -2,6 +2,7 @@ - **Status:** Accepted - **Date:** 2026-10-03 +- **Area:** — - **Source:** Squad adopted from Squad-Spec-Repository-Template - **Supersedes:** —