From b30486c8958c758da9dd7f475300d16a6bd9d03e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:38:29 +0000 Subject: [PATCH 01/12] Add the area index and the Area field of the decision records Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/README.md | 11 +++++++++++ .../0001-quality-gates-before-the-pull-request.md | 1 + docs/decisions/0002-squash-merge-pull-requests.md | 1 + docs/decisions/0003-squad-working-records-off-main.md | 1 + 4 files changed, 14 insertions(+) diff --git a/docs/areas/README.md b/docs/areas/README.md index 7151574..61e31ab 100644 --- a/docs/areas/README.md +++ b/docs/areas/README.md @@ -27,4 +27,15 @@ operator or another component can rely on — formats, limits, error behavior, g | Area | Scope | Not here | | ---- | ----- | -------- | +| [Hosting and configuration](hosting.md) | Loading and validating configuration, startup sequence and database migrations, HTTP pipeline and error pages, container image and runtime user, security posture | Meaning of feature-specific keys (their areas); full key reference (`README.md`); build, CI and release (`CONTRIBUTING.md`) | +| [Docker instances](docker-instances.md) | Docker Engine connectivity and transports, certificate relaxation, discovery of running containers and their runtime status, resource statistics | Update evaluation of containers (Update detection); Portainer actions (Container actions); schedules (Scan engine) | +| [Registries](registries.md) | Read access to Docker Hub and OCI registries, registry dispatch, Docker Hub account discovery and request budget, retry of transient HTTP failures, failure model of all integrations | When scans run (Scan engine); Docker Scout (Vulnerabilities); base-image resolution (Base images) | +| [Image catalog](image-catalog.md) | Image reference parsing and normalization, the deduplicated registry/repository/tag/digest catalog, observed images and how they are registered | Base images (Base images); tag comparison (Update detection); CVEs (Vulnerabilities) | +| [Base images](base-images.md) | Base-image resolution, the base-image relationship graph, shared base images, derived .NET/nginx runtime detection against release feeds | Tag ranking (Update detection); registry access (Registries) | +| [Update detection](update-detection.md) | Tag formats and variant families, alias resolution, pre-release ranking, major-upgrade gating, update assessment, findings and tag candidates, manual tag selection | Fetching tags (Registries); retention (Scan engine) | +| [Vulnerabilities](vulnerabilities.md) | Provider choice (Trivy, Docker Scout, none), assessment status, severities, fix status and CVSS, manual rescan, configuration hints | Schedule and cleanup (Scan engine); display (Web UI) | +| [Container actions](container-actions.md) | Portainer endpoints and authentication, insecure HTTP opt-in, restart/redeploy/update actions and their recorded runs | Container discovery (Docker instances) | +| [Scan engine](scan-engine.md) | Background jobs and their intervals, dynamic own-image interval, overlap rules, scan runs, crash recovery, retention and cleanup, single-instance assumption | What a scan evaluates (the feature areas) | +| [Web UI](web-ui.md) | Pages and routes, what each page shows and triggers, dashboard refresh, error and not-found pages, theme | How the shown values are computed (the feature areas) | +| [Telemetry](telemetry.md) | Logs, traces and metrics, OTLP export, activity and instrument names, inventory gauges, what never reaches telemetry | Deployment of the observability stack (`INSTALL.md`) | diff --git a/docs/decisions/0001-quality-gates-before-the-pull-request.md b/docs/decisions/0001-quality-gates-before-the-pull-request.md index 006e0b9..942b0e6 100644 --- a/docs/decisions/0001-quality-gates-before-the-pull-request.md +++ b/docs/decisions/0001-quality-gates-before-the-pull-request.md @@ -2,6 +2,7 @@ - **Status:** Accepted - **Date:** 2026-10-03 +- **Area:** — - **Source:** Squad adopted from Squad-Spec-Repository-Template - **Supersedes:** — diff --git a/docs/decisions/0002-squash-merge-pull-requests.md b/docs/decisions/0002-squash-merge-pull-requests.md index 972befc..2df5731 100644 --- a/docs/decisions/0002-squash-merge-pull-requests.md +++ b/docs/decisions/0002-squash-merge-pull-requests.md @@ -2,6 +2,7 @@ - **Status:** Accepted - **Date:** 2026-10-03 +- **Area:** — - **Source:** Squad adopted from Squad-Spec-Repository-Template - **Supersedes:** — diff --git a/docs/decisions/0003-squad-working-records-off-main.md b/docs/decisions/0003-squad-working-records-off-main.md index ba57acc..539bd26 100644 --- a/docs/decisions/0003-squad-working-records-off-main.md +++ b/docs/decisions/0003-squad-working-records-off-main.md @@ -2,6 +2,7 @@ - **Status:** Accepted - **Date:** 2026-10-03 +- **Area:** — - **Source:** Squad adopted from Squad-Spec-Repository-Template - **Supersedes:** — From b3735d8780873d66e70cd71422e9c12f8b127a39 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:41:35 +0000 Subject: [PATCH 02/12] Describe the Docker instances and container actions areas Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/container-actions.md | 117 +++++++++++++++++ docs/areas/docker-instances.md | 180 ++++++++++++++++++++++++++ docs/areas/hosting.md | 185 +++++++++++++++++++++++++++ docs/areas/scan-engine.md | 215 ++++++++++++++++++++++++++++++++ 4 files changed, 697 insertions(+) create mode 100644 docs/areas/container-actions.md create mode 100644 docs/areas/docker-instances.md create mode 100644 docs/areas/hosting.md create mode 100644 docs/areas/scan-engine.md diff --git a/docs/areas/container-actions.md b/docs/areas/container-actions.md new file mode 100644 index 0000000..c4bf942 --- /dev/null +++ b/docs/areas/container-actions.md @@ -0,0 +1,117 @@ +# Container actions + +## Scope + +The optional Portainer integration of a Docker instance: how a Portainer server is configured and mapped to a Docker +instance, how DockerUpdateGuard authenticates against it, which container actions it can execute through it, and the +stored records reserved for action runs. Operators rely on it to configure Portainer access without weakening +transport security; the UI is the intended caller of the actions. + +## Behavior + +### Configuration and mapping + +- Portainer is configured per Docker instance under `DockerUpdateGuard:DockerInstances::Portainer` with the keys + `Enabled` (default `false`), `BaseUrl`, `Username`, `Password`, `ApiToken`, `EndpointId`, `AllowInsecureHttp` + (default `false`) and `RequestTimeoutSeconds` (default `15`). +- A Docker instance has at most one Portainer endpoint. It is synchronized together with the instance + ([Docker instances](docker-instances.md)): + - when Portainer is enabled, the stored endpoint is created or updated with the name ` Portainer`, + the trimmed `BaseUrl` and the configured `EndpointId`, and marked enabled; + - when Portainer is disabled, an existing stored endpoint is kept but marked disabled; + - when the Docker instance is removed from configuration, its Portainer endpoint and action runs are deleted with it. +- Credentials (`Username`, `Password`, `ApiToken`) are never stored in the database; they are read from configuration + at the time of each call. +- `EndpointId` selects the Portainer environment that manages the Docker instance. When it is empty, the first + environment returned by Portainer's environment list is used, resolved again on every call. + +### Transport security + +- `BaseUrl` must be an absolute `https` URL. A plaintext `http` URL is accepted only with `AllowInsecureHttp: true`; + without it, startup fails with a message that names the key to set ([Hosting](hosting.md)). +- `AllowInsecureHttp` is opt-in only. It is a validation switch: no runtime warning is logged while it is active. +- Portainer server certificates are always validated; there is no option to skip validation for Portainer. + +### Authentication + +- When `ApiToken` is set, it is sent as an API key header on every request and takes precedence over username and + password. +- Otherwise `Username` and `Password` are posted to Portainer's authentication endpoint and the returned JWT is used as + a bearer token. The login is repeated for every operation; tokens are not cached. +- A login that answers non-success, or answers success without a token, ends the operation with an authentication + error that names the Portainer base URL and the HTTP status code; credentials never appear in it or in the log. + +### Capability check + +- For a Docker instance, a capability check reports whether Portainer is configured, whether actions are supported, and + a message: + - Portainer disabled or without `BaseUrl` → not configured, no actions; + - Portainer's status endpoint answers non-success → configured, no actions, message with the HTTP status code; + - no environment configured and none found → configured, no actions, "No Portainer endpoint found"; + - otherwise → configured, actions supported, message naming the environment used; + - any error (including authentication failure) → configured, no actions, message with the error text. + +### Actions + +- The executable actions are container actions only: `start`, `stop`, `restart`, `kill`, `pause` and `unpause` + (case-insensitive). Any other action name is rejected before Portainer is contacted, with a message listing the + allowed names. +- The target container is named, not identified by ID: Portainer's container list for the environment (including + stopped containers) is searched for an exact, case-insensitive name match, with or without the leading `/`. +- The action is executed by Portainer on the Docker Engine of that environment. The result is either *succeeded* with + a confirmation message or *failed* with a reason: Portainer not configured, no environment found, container not + found, HTTP status code of a non-success answer, or the error text. +- Actions never throw to the caller; every outcome is a result. Executed, failed and not-found actions are logged with + the action, the container name and the Docker instance name. +- Portainer requests go through the shared retry of transient HTTP failures described in + [Registries](registries.md). This includes the login and the action request itself. +- `RequestTimeoutSeconds` (1–300, default 15) bounds each Portainer request including its retries. + +### Action runs + +- The data model holds an action-run record per requested action: the Docker instance, optionally the Portainer + endpoint and the container snapshot, the action type (*restart container*, *redeploy stack*, *update service*), the + resource type (*container*, *service*, *stack*), the resource name (up to 200 characters), status (*requested*, + *running*, *succeeded*, *failed*, *cancelled*), requester, Portainer task ID, error message, and request and + completion times. +- A container snapshot referenced by an action run is never removed by retention ([Scan engine](scan-engine.md)). + Deleting the Portainer endpoint or the snapshot keeps the action run and clears the link. +- Today no component executes actions or writes action runs: neither a page nor a background job calls the capability + check or an action (see *Open questions*). + +## Limits and errors + +- Validation, applied only when `Enabled` is `true`, fails startup when: `BaseUrl` is missing; `BaseUrl` is not an + absolute `http`/`https` URI; `BaseUrl` is `http` without `AllowInsecureHttp: true`; neither `ApiToken` nor both + `Username` and `Password` are set; `RequestTimeoutSeconds` is outside 1–300. +- A disabled Portainer section is not validated at all; its values may be incomplete. +- Error messages and log entries name the Docker instance, the Portainer base URL, the action, the container name and + HTTP status codes. They never contain the password, the API token or the JWT. + +## Related decisions + +None yet. + +## Not here + +- Container discovery, snapshots and Docker Engine connectivity: [Docker instances](docker-instances.md). +- The retry policy and the shared failure model of outbound HTTP: [Registries](registries.md). +- Startup validation failure and configuration reload: [Hosting](hosting.md). +- Retention of snapshots and other history: [Scan engine](scan-engine.md). +- What the UI shows about Portainer availability: [Web UI](web-ui.md). + +## Open questions + +1. No page or job triggers Portainer actions, and no action run is ever written. Is the integration meant to be wired + into the UI (the security area list names "the UI that triggers them"), or should the unused parts be removed? +2. The executable actions (`start`, `stop`, `restart`, `kill`, `pause`, `unpause` on containers) do not match the + action types of the action-run record (*restart container*, *redeploy stack*, *update service* on containers, + services and stacks). Which set is the intended contract? Redeploying stacks and updating services are not + implemented. +3. The action request and the login are retried on HTTP 408, 429 and 5xx answers like any idempotent request, so one + requested action can be executed more than once. Should non-idempotent Portainer requests be excluded from retry? +4. `AllowInsecureHttp` produces no runtime warning, while the documentation of the security posture says it is logged + as a warning at runtime. Should a warning be added, as for `SkipCertificateValidation`? +5. Without `EndpointId`, the first environment Portainer lists is used, which can be an environment other than the one + that manages the Docker instance. Should an explicit `EndpointId` be required when Portainer manages more than one + environment? diff --git a/docs/areas/docker-instances.md b/docs/areas/docker-instances.md new file mode 100644 index 0000000..0480766 --- /dev/null +++ b/docs/areas/docker-instances.md @@ -0,0 +1,180 @@ +# Docker instances + +## Scope + +How DockerUpdateGuard connects to the Docker Engines it watches: the configured instances and their transports, the +read-only Engine calls it makes, the discovery of containers and their runtime status, the container snapshots that +record what ran where, and the CPU, memory and network samples collected per container and per instance. Operators +rely on it to configure engine access safely; the scan engine, update detection, base-image resolution and the UI rely +on the snapshots and samples it produces. + +## Behavior + +### Configured instances + +- Docker instances exist only through configuration: each entry of `DockerUpdateGuard:DockerInstances` is one + instance. There is no way to add an instance from the UI. +- Per instance the keys are `Name` (required), `BaseUrl` (required), `Enabled` (default `true`), `UseTls` (default + `false`), `SkipCertificateValidation` (default `false`), `CertificatePath` (optional), `RequestTimeoutSeconds` + (default `15`) and the optional `Portainer` section (see [Container actions](container-actions.md)). +- An instance is identified by its trimmed `Name`, compared case-insensitively. Renaming an instance in configuration + is the removal of the old instance and the creation of a new one. +- The configured instances are synchronized into the database at startup, on every + `DockerUpdateGuard:Scanning:DiscoveryIntervalMinutes` tick, and at the start of every runtime container scan and + every resource-statistics collection (schedules: [Scan engine](scan-engine.md)). A synchronization: + - creates a record for each new name, with registration source *configuration file*; + - updates endpoint URI (trimmed `BaseUrl`), connection kind, enabled flag and `SkipCertificateValidation` of each + existing one; + - **deletes** every stored instance whose name is no longer configured, together with its scan runs, container + snapshots, resource samples, Portainer endpoint and action runs. Update findings that pointed at a deleted + snapshot lose that link but are kept. +- A disabled instance (`Enabled: false`) keeps its stored history but is not contacted: no discovery, no inspection, + no resource sampling. Every Engine call for it answers *not configured*. +- The stored connection kind is derived from `BaseUrl`: `http` → Http, `https` → Https, `tcp` → Https when `UseTls` + is `true`, otherwise Http, `unix` → UnixSocket, `npipe` → NamedPipe. Any other value is stored as *not set* and an + enabled instance with it is logged as a warning. + +### Transports + +- `http://` and `https://` URLs are used as the Engine base address as given. +- `tcp://host:port` is turned into `https://host:port` when `UseTls` is `true`, otherwise `http://host:port`. + `UseTls` has no effect on any other scheme. +- `unix:///path/to/socket` connects to that Unix domain socket (for example `unix:///var/run/docker.sock`). +- `npipe://server/pipe/name` connects to a Windows named pipe; an empty host or `.` means the local machine, a leading + `pipe/` segment is optional. A named-pipe URL without a pipe name fails every call for that instance. +- Engine connections never use an HTTP proxy. Connections are pooled per instance and per connection-relevant + setting (name, `BaseUrl`, TLS, certificate validation, certificate path); pooled TCP connections are renewed after + at most 5 minutes. +- Engine requests use the Docker Engine API version `v1.41` path prefix. +- Engine requests are not retried; a failed request fails that operation (the shared retry of outbound HTTP described + in [Registries](registries.md) does not apply to Engine connections). + +### TLS options + +- `SkipCertificateValidation: true` accepts any server certificate on `https` and TLS `tcp` connections. It is opt-in + only. Whenever it is active, a warning saying that the connection is unauthenticated and open to man-in-the-middle + attacks is logged once per instance name per process lifetime, when the first connection for that instance is + created. +- `CertificatePath` names a client certificate presented on `https` and TLS `tcp` connections. A `.pfx` or `.p12` + file is loaded as a password-less PKCS#12 bundle; any other extension is loaded as a single certificate file. A path + that does not exist is ignored without a warning. +- Neither option has an effect on `unix` and `npipe` connections. + +### Timeouts + +- `RequestTimeoutSeconds` (1–300, default 15) bounds each single Engine request (container list, each image inspect, + each image history, each stats call, host info) separately; it is not a budget for a whole scan. +- A request that exceeds it ends as *failed* with a message naming the instance and the timeout. Cancellation of the + calling job (for example at shutdown) is not reported as a failure; it stops the operation. + +### Container discovery + +- Discovery lists **all** containers of an instance, including stopped ones. +- Per container it records: the container ID; the name (the first listed name without its leading `/`, or the + container ID when it has no name); the image reference as the Engine reports it; the local image ID; the compose + project (`com.docker.compose.project` label), the stack name (`com.docker.stack.namespace`) and the swarm service + name (`com.docker.swarm.service.name`). +- The runtime status is one of Running, Paused, Restarting, Exited, Dead; any other Engine state (for example + `created`) is *not set*. A container counts as running only in state `running`. +- The image digest of each container is taken from the repository digests of its local image (one image inspect per + distinct local image): the digest whose registry and repository match the container's image reference; if none + matches, the only digest when there is exactly one; otherwise none. When the image reference cannot be parsed, the + first repository digest is used. Digests are stored lower-case. A failed inspect leaves the digest empty and does + not fail discovery. +- Discovery emits one `dockerengine.request` trace activity per instance (see [Telemetry](telemetry.md)). + +### Image inspection and history + +- Other areas can read the inspect data of a local image (ID, repo tags, repo digests, environment variables, root + file-system layers, creation time, operating system, architecture) and its history (creation time, created-by + command, comment, tags per layer) from an instance, addressed by image ID or reference. +- An image the Engine does not know answers *not found*; any other non-success answer *failed*. How the data is used + belongs to [Base images](base-images.md) and [Update detection](update-detection.md). + +### Container snapshots + +- Each runtime container scan of an enabled instance records one scan run for the instance and one container snapshot + per discovered container, linked to that scan run. Snapshots are append-only history: every scan adds new rows, + earlier ones stay until retention removes them ([Scan engine](scan-engine.md)). +- A snapshot holds the instance, container ID, name, compose project, stack name, service name, runtime status, + running flag, the catalogued image version (registry, repository, tag and the discovered digest, see + [Image catalog](image-catalog.md)) and its recording time. It starts with update assessment *not evaluated*; the + assessment fields are filled by [Update detection](update-detection.md). +- The **current** containers of an instance are the snapshots of its most recent runtime scan run that recorded + snapshots. A later run that recorded none (for example because the Engine was unreachable) does not empty the + current set; the previous set stays current. +- A container whose image reference cannot be parsed produces no snapshot; the scan run then ends *partial* with a + message naming the container. +- Outcome of the discovery step for the scan run: a *failed* discovery ends the run *failed*; *not configured* or + *unsupported* ends it *partial*; an unexpected error ends it *failed*. The run's error message joins the distinct + messages of all problems. One failing instance never stops the scans of the other instances. +- Configured instances that are disabled, or stored instances that are missing from the current configuration, are + skipped by the scan without a scan run. + +### Resource statistics + +- Every `DockerUpdateGuard:Scanning:ResourceStatisticsIntervalMinutes` (1–1440, default 5) each enabled, configured + instance is sampled ([Scan engine](scan-engine.md) owns the schedule). +- Only **running** containers are sampled, one at a time, each with a single non-streaming stats request. A container + whose stats request answers non-success is left out of that round. +- Per container sample: + - CPU percent = (container CPU time delta ÷ system CPU time delta) × online CPUs × 100, rounded to 4 decimals; the + CPU count falls back to the per-CPU usage list length and then to 1; 0 when either delta is not positive. A + multi-core container can exceed 100. + - Memory usage and memory limit are the Engine's reported usage and limit in bytes. + - Network receive and transmit totals are the sums over all of the container's network interfaces, in bytes. + - The recording time is the Engine's stats timestamp, or the current time when it is missing. + - Receive and transmit rates in bytes per second are computed against the previous sample of the same container ID + on the same instance, rounded to 4 decimals. The rate is 0 for the first sample, when no time has passed, or when + the total did not grow (for example after a container restart reset the counter). +- Per instance, one sample per round: container count, CPU percent and memory usage as the sums over the container + samples, memory limit as the host's total memory reported by the Engine (0 when that request fails), network totals + as sums, rates against the previous instance sample by the same rule, and the latest container recording time (the + current time when no container was sampled). An instance with no running containers still gets a sample with + count 0. +- Samples are kept until retention removes them ([Scan engine](scan-engine.md)); how they are charted belongs to + [Web UI](web-ui.md). + +## Limits and errors + +- Invalid instance configuration fails startup ([Hosting](hosting.md)): a missing `Name`, a duplicate name (after + trimming, case-insensitive), a missing `BaseUrl`, a `BaseUrl` that is not an absolute URI with scheme `http`, + `https`, `tcp`, `unix` or `npipe`, or a `RequestTimeoutSeconds` outside 1–300. The shipped `appsettings.json` + contains one instance with an empty `Name`, so an operator must set a name before the application starts. +- Every Engine operation reports its outcome as a status (*succeeded*, *not configured*, *unsupported*, *not found*, + *failed*) instead of throwing; the shared failure model is described in [Registries](registries.md). +- A non-success Engine answer is reported as *failed* with the instance name, the HTTP status code and the response + body. Unexpected errors are reported with the instance name and the error message. These messages can end up in a + scan run's error message and are shown in the UI. +- A resource collection that fails for an instance (list request failed or timed out, or any stats request timed out + or broke) records **no** samples for that instance in that round and logs a warning; other instances are not + affected. +- `SkipCertificateValidation` and a mounted Docker socket are trust relaxations the operator chooses; the socket is + root-equivalent on the host ([Hosting](hosting.md)). + +## Related decisions + +None yet. + +## Not here + +- When scans and sampling run, retention of snapshots, samples and scan runs: [Scan engine](scan-engine.md). +- How a container's image is evaluated for updates and the assessment statuses: [Update detection](update-detection.md). +- Base-image and derived-runtime detection from inspect and history data: [Base images](base-images.md). +- Image reference parsing and the image catalog: [Image catalog](image-catalog.md). +- Restart and other actions through Portainer: [Container actions](container-actions.md). +- Pages that show instances, containers and resource charts: [Web UI](web-ui.md). +- Startup, validation failure and the container image's socket access: [Hosting](hosting.md). + +## Open questions + +1. Instance lookups during a scan or a resource collection match the configured `Name` untrimmed, while the stored + name is trimmed. A configured name with leading or trailing spaces is therefore stored but always skipped as + "missing from configuration". Is that intended, or should both sides trim? +2. A `CertificatePath` that is not a `.pfx`/`.p12` file is loaded as a certificate without a private key, which + cannot authenticate a TLS client. Should PEM certificate-plus-key files be supported, or only PKCS#12 be accepted? +3. A missing `CertificatePath` file is ignored silently. Should it fail validation or log a warning? +4. A single stats timeout discards the whole round of samples for an instance, while a non-success stats answer only + drops that container. Is the asymmetry intended? +5. Removing an instance from configuration deletes its whole history, including scan runs, without retention. Is + that the desired behavior, or should a removed instance be kept as disabled? diff --git a/docs/areas/hosting.md b/docs/areas/hosting.md new file mode 100644 index 0000000..f79065e --- /dev/null +++ b/docs/areas/hosting.md @@ -0,0 +1,185 @@ +# Hosting and configuration + +## Scope + +How the application is configured, how it starts, how it serves HTTP and how its container image is built to run. +It covers loading and validating configuration, resolving the database connection, the startup sequence up to the +point where requests are served, configuration reload, the HTTP-level pipeline and error pages, the container image +and the security posture the deployment must provide. Operators rely on it to deploy and configure the application; +every feature area relies on it for validated configuration and a migrated database. + +## Behavior + +### Configuration sources + +- Configuration is read from, in increasing precedence: `appsettings.json`, `appsettings.{Environment}.json`, user + secrets (Development only), environment variables and command-line arguments. Environment variables use `__` as the + section separator (`DockerUpdateGuard__Scanning__RetryCount`). +- The application reads four top-level sections: `DockerUpdateGuard` (application settings), `Telemetry` (see + [Telemetry](telemetry.md)), `ConnectionStrings` and `Logging`. The key-by-key reference with defaults and ranges is + the configuration reference in `README.md`; what each feature key means is stated in the feature's area document. +- The environment defaults to `Production`. Only `Development` changes behavior here (see *HTTP pipeline*). +- Secrets (database password, Docker Hub PAT, Portainer password or API token) are ordinary configuration values; + any of the sources above may supply them. There is no further secret-store integration. + +### Validation + +- The whole `DockerUpdateGuard` section is validated as one unit. All rule violations are collected and reported + together, each naming the offending key (for example `'DockerUpdateGuard:Database:MigrationRetryDelaySeconds' must be + greater than zero`). Messages name keys and, at most, configured instance names; they never echo connection + strings, passwords or tokens. +- Any violation fails startup: the process exits before the database is contacted and before any request is served. +- The `Telemetry` section is validated even earlier, while services are registered; see [Telemetry](telemetry.md). +- Rules owned by this area: + - a connection string must resolve to a non-blank value (see *Database connection*); + - `DockerUpdateGuard:Database:MaxConnectionRetryCount`, `MaxConnectionRetryDelaySeconds`, + `MigrationStartupTimeoutSeconds` and `MigrationRetryDelaySeconds` must each be greater than zero. +- The rules for feature keys (Docker Hub, release metadata, vulnerabilities, scanning, Docker instances, Portainer) + are part of the respective area documents; they are enforced by the same validation step. + +### Database connection + +- The database is PostgreSQL. The connection string is resolved in this order: + 1. `DockerUpdateGuard:ConnectionString`, trimmed, when it is not blank; + 2. otherwise `ConnectionStrings:{name}`, where `{name}` is `DockerUpdateGuard:ConnectionStringName` + (default `DockerUpdateGuard`). +- Validation and the database registration use the same resolution, so the validated connection string is always + the one in use. +- During normal operation, a database command that fails transiently is retried up to + `DockerUpdateGuard:Database:MaxConnectionRetryCount` times (default `8`) with a back-off of at most + `DockerUpdateGuard:Database:MaxConnectionRetryDelaySeconds` seconds (default `30`) between attempts. + +### Startup sequence + +Startup runs these steps in order; requests are served and background jobs start only after all of them have +succeeded. + +1. **Configuration** is bound and validated (see *Validation*). +2. **Wait for the database.** The application opens and closes a connection. A transient failure (a database error + the driver reports as transient, a socket error or a timeout, also when wrapped in another error) is retried every + `DockerUpdateGuard:Database:MigrationRetryDelaySeconds` seconds (default `5`) until + `DockerUpdateGuard:Database:MigrationStartupTimeoutSeconds` (default `120`) have elapsed since the first attempt. + Each retry is logged with its attempt number. A non-transient failure, or a transient one after the deadline, is + logged and fails startup. A shutdown request during the wait ends it. +3. **Apply migrations.** Pending schema migrations are applied automatically. On PostgreSQL the application first + takes the session-scoped advisory lock with key `5410320540103205` on the same connection that runs the + migrations, so instances starting concurrently against one database apply migrations one at a time; the others + wait for the lock and then find nothing left to apply. Taking the lock and migrating form one unit: if a transient + fault forces a retry, both are repeated together. The lock is released afterwards; a failed release is logged and + does not change the outcome, because the lock also ends with the session. A failed migration is logged and fails + startup. +4. **Synchronize configured Docker instances** into the database (see [Docker instances](docker-instances.md)). +5. **Seed the inventory gauges** from the database (see [Telemetry](telemetry.md)). + +Only local database work runs on this path. Work that needs the network, such as Docker Hub account image +discovery, runs in the background jobs after startup (see [Scan engine](scan-engine.md)), so an unreachable +registry never delays or prevents startup. + +### Configuration reload + +- `appsettings.json` and `appsettings.{Environment}.json` are watched; a change is picked up without a restart. + Values that are read on use take effect at their next use; the feature areas state which of their keys do, and the + [Scan engine](scan-engine.md) states when job intervals change. +- These values are read once at startup and need a restart to change: + - the connection string, `DockerUpdateGuard:ConnectionStringName` and all `DockerUpdateGuard:Database:*` keys; + - `DockerUpdateGuard:DockerHub:ApiBaseUrl`, `DockerUpdateGuard:DockerHub:RequestTimeoutSeconds` (also the request + timeout for generic registries and release-metadata feeds) and `DockerUpdateGuard:ReleaseMetadata:*`; + - the whole `Telemetry` section. +- A reloaded configuration is validated by the same rules as at startup when it is next read; see *Open questions*. + +### HTTP pipeline + +- The application serves only its interactive web UI and static assets; there is no separate HTTP API. +- Outside `Development`: + - an unhandled exception while processing a request renders the `/error` page with a generic message + ("Unexpected error"); no exception details, stack traces or configuration values are shown; + - HSTS is applied and HTTP requests are redirected to HTTPS when an HTTPS port is known to the server. +- In `Development`, the platform's developer exception page is shown instead and there is no HSTS or HTTPS + redirection. +- In every environment, a response with an error status code and no body (such as `404`) is re-rendered as the + `/not-found` page, and an unknown route inside the UI shows the same page. Both pages are described in + [Web UI](web-ui.md). +- Form posts are protected by anti-forgery validation. + +### Container image + +- The runtime image is based on the Alpine variant of the ASP.NET Core runtime (`mcr.microsoft.com/dotnet/aspnet`, + `10.0-alpine` by default). The base image name and digest are recorded as the OCI labels + `org.opencontainers.image.base.name` and `org.opencontainers.image.base.digest`. +- The application listens on plain HTTP on port `8080` on all interfaces. TLS is not terminated inside the container. +- The process runs as UID `64000` in the root group (GID `0`), never as UID `0`. `/app` is owned by `64000:0` with + group permissions equal to owner permissions, so the application also works when started with another UID in + group `0`. Membership in group `0` keeps a `root:root` Docker socket readable where the socket grants group access. +- The Trivy CLI is bundled at `/usr/local/bin/trivy` (taken from `aquasec/trivy:0.58.1` by default) with its cache in + `/tmp/trivy`, so Trivy client mode works without extra installation (see [Vulnerabilities](vulnerabilities.md)). +- The image sets `DockerUpdateGuard__DisplayVersion` to the version it was built with. +- CA certificates are installed so outbound TLS connections can be verified. +- Before the application starts, the entrypoint: + 1. checks that `/app/appsettings.json` and `/app/appsettings.{Environment}.json`, if present, are readable by the + running user; if one exists but is not readable, it prints the UID/GID and how to fix the host permissions, + and exits with code `1`; + 2. if `/app/certs` contains files, imports every `.crt` and `.pem` file into the system trust store, but only when + running as root or when the trust store is writable; otherwise, or when the import fails, it prints a message + and continues. Certificate import never stops the container. + +### Security posture + +These properties hold today and must be provided for by the deployment: + +- **No authentication or authorization.** Every page and every action (manual scans, tag selection, container + actions) is available to anyone who can reach port `8080`. The application must run behind a trusted perimeter: a + private network, a VPN or a reverse proxy that authenticates users. +- **No health-check endpoint and no rate limiting.** +- **Docker socket access is root-equivalent on the host.** Mounting the host's Docker socket gives the application, + and anyone who controls it, the power of root on that host. This is inherent to the feature. +- **Single instance.** Several instances may share one database for migrations (see *Startup sequence*), but the + background jobs do not coordinate across instances (see [Scan engine](scan-engine.md)). +- That secrets never appear in logs or telemetry is stated in [Telemetry](telemetry.md); that the UI never shows + them is stated in [Web UI](web-ui.md). + +## Limits and errors + +| Situation | Result | +| --- | --- | +| Invalid `DockerUpdateGuard` configuration at startup | Startup fails; every violation is reported with its key. | +| No connection string resolves | Startup fails: `'DockerUpdateGuard:ConnectionString' or 'ConnectionStrings:{name}' must be configured`. | +| Invalid `Telemetry` configuration | Startup fails before the host is built (see [Telemetry](telemetry.md)). | +| Database unreachable within `MigrationStartupTimeoutSeconds` | The attempts are logged; startup fails. | +| Non-transient database error while waiting | Startup fails immediately. | +| Migration fails | Logged; startup fails. A schema is never half-reported as ready. | +| Advisory lock cannot be released | Logged; the migration outcome stands. | +| Mounted settings file not readable by the container user | Entrypoint exits with code `1` and a fix-it message. | +| Certificate import not possible | A message is printed; the container starts anyway. | +| Unhandled exception in a request (non-Development) | The `/error` page with a generic message. | +| Unknown URL or bodyless error status | The `/not-found` page. | + +## Related decisions + +None yet. + +## Not here + +- What each feature key means and its valid range: the feature areas ([Docker instances](docker-instances.md), + [Registries](registries.md), [Base images](base-images.md), [Update detection](update-detection.md), + [Vulnerabilities](vulnerabilities.md), [Container actions](container-actions.md), [Scan engine](scan-engine.md)). +- The complete configuration key reference: `README.md`, *Configuration reference*. +- Retry of outbound HTTP calls: [Registries](registries.md). +- Logs, traces, metrics and the rule that credentials stay out of them: [Telemetry](telemetry.md). +- Pages and what they show: [Web UI](web-ui.md). +- Building, testing, CI and releasing the image: `docs/CONTRIBUTING.md`. +- Compose files and the walkthrough for the deployment tiers: `INSTALL.md`. + +## Open questions + +1. Background jobs read their interval from the reloaded configuration outside their error handling. If a reload + produces a configuration that fails validation, reading it raises an error there that is not caught, which can + stop the host instead of keeping the last valid configuration. Is stopping the intended reaction to an invalid + reload? +2. There is no handling of forwarded headers from a reverse proxy. Behind a TLS-terminating proxy the application + sees plain HTTP, so HSTS is never sent and HTTPS redirection cannot determine a port. Is relying on the proxy for + both intended? +3. In the shipped image the trust store is owned by root and not writable by group `0`, so the certificate import + from `/app/certs` appears to be skipped unless the container is started as root. Should the import work for the + default user? +4. An instance waiting for the migration advisory lock is subject to the database command timeout. Whether a long + migration by one instance can make a concurrently starting instance fail its startup is not established. diff --git a/docs/areas/scan-engine.md b/docs/areas/scan-engine.md new file mode 100644 index 0000000..7616570 --- /dev/null +++ b/docs/areas/scan-engine.md @@ -0,0 +1,215 @@ +# Scan engine + +## Scope + +When and how often the application refreshes its data in the background, how each refresh is recorded as a scan +run, how runs that never finished are repaired and how old scan data is removed. Operators rely on it through the +`DockerUpdateGuard:Scanning` interval and retention keys; the [Web UI](web-ui.md) relies on the scan-run records +for its scan history and status chips. What a single scan evaluates belongs to the feature areas listed under +*Not here*. + +## Behavior + +### Background jobs + +The host runs exactly seven recurring background jobs, all in the application process: + +| Job | What it triggers | Interval key (`DockerUpdateGuard:Scanning:…`) | Default | Allowed range | +| --- | --- | --- | --- | --- | +| Docker instance synchronization | Reconcile the configured Docker instances with the stored ones ([Docker instances](docker-instances.md)) | `DiscoveryIntervalMinutes` | 15 min | 1–1440 | +| Docker Hub account discovery | Refresh the images of the configured Docker Hub account ([Registries](registries.md)) | `DockerHubAccountDiscoveryIntervalMinutes` | 60 min | 1–10080 | +| Observed-image refresh | Scan every enabled observed image (tag metadata, base images, derived runtime) | dynamic, see below (`OwnImageBaseScanIntervalMinutes` is the floor) | 60 min | 1–10080 | +| Runtime container refresh | Scan the running containers of every enabled Docker instance | `RuntimeImageUpdateScanIntervalMinutes` | 30 min | 1–1440 | +| Resource statistics | Sample CPU, memory and network usage ([Docker instances](docker-instances.md)) | `ResourceStatisticsIntervalMinutes` | 5 min | 1–1440 | +| Vulnerability refresh | Re-assess the vulnerabilities of all live images ([Vulnerabilities](vulnerabilities.md)) | `VulnerabilityRefreshIntervalMinutes` | 180 min | 1–10080 | +| Scan cleanup | Repair unfinished runs and purge old data (see *Retention and cleanup*) | `CleanupIntervalMinutes` | 720 min | 1–10080 | + +- The jobs start after the startup sequence of [Hosting and configuration](hosting.md) has completed (database + reachable, migrations applied). +- Every job except scan cleanup runs once immediately when it starts and then waits one interval between runs. + Scan cleanup first performs its stale-run repair (see *Unfinished runs*) and runs its first full pass only after + one `CleanupIntervalMinutes` interval. +- The wait is measured from the end of one run to the start of the next, so runs of one job never overlap each + other and a long run delays the next one. +- The jobs are independent of each other: they are neither ordered nor coordinated, and two different jobs may run + at the same time. +- The interval is read again before every wait. A changed interval key takes effect from the next wait without a + restart; a wait already in progress is not shortened or extended. +- Each run works in its own unit of work (its own database context), so no state carries over between runs. + +### Dynamic observed-image interval + +The observed-image refresh stretches its interval with the number of enabled observed images so that scheduled +Docker Hub requests stay within a request budget: + +``` +budget = DockerHubRequestLimitPerWindow − DockerHubReservedManualRequestsPerWindow +interval = max(OwnImageBaseScanIntervalMinutes, ⌈DockerHubRequestLimitWindowHours × 60 × enabledImages / budget⌉) minutes +``` + +- With no enabled observed image the interval is `OwnImageBaseScanIntervalMinutes`. +- The count of enabled observed images is read again before every wait. +- Defaults and ranges: `DockerHubRequestLimitWindowHours` 6 (1–168), `DockerHubRequestLimitPerWindow` 200 + (1–100000), `DockerHubReservedManualRequestsPerWindow` 40 (0–100000). The reserved share must be smaller than the + limit per window, so the budget is always at least 1. +- With the defaults, up to 26 enabled images are scanned every 60 minutes; from 27 images on the interval grows by + about 2.25 minutes per image (e.g. 100 images → 225 minutes). +- The formula only spaces out scheduled runs. Manual scans are not counted against the budget, and the job does not + count the requests a run actually makes. Request parallelism and HTTP 429 handling belong to + [Registries](registries.md). + +### Scan runs + +Three kinds of work are recorded as scan runs; the other jobs (instance synchronization, account discovery, +resource statistics, cleanup) leave no scan-run record and report only through logs. + +| Type | One run per | Started by | +| --- | --- | --- | +| `ObservedImage` | observed image scanned | the observed-image refresh (one run per enabled image), or registering an image in the UI | +| `RuntimeContainer` | enabled Docker instance scanned | the runtime container refresh, or the manual update check in the UI | +| `Vulnerability` | vulnerability refresh | the vulnerability refresh (one run for all live images), or a manual refresh or per-image rescan in the UI | + +- A run records its type, status, trigger, start time, completion time, an optional error message and the observed + image or Docker instance it concerns. +- Trigger is `Scheduled` for runs started by a background job and `Manual` for runs started from the UI. The stored + values `Startup` and `Discovery` are reserved and not written today. +- A run is stored with status `Running` before any work begins and is updated once at the end to one of: + - `Succeeded` — every step completed; + - `Partial` — at least one step returned an incomplete result (for example an unsupported registry or one + container that could not be processed) while the run as a whole went on; + - `Failed` — a step reported a failure, or an unexpected error ended the run. +- The error message joins the distinct messages of all incomplete or failed steps, one per line; it is empty for a + `Succeeded` run. +- The stored values `NotSet` and `Pending` are reserved and not written today. +- A vulnerability refresh that is disabled by configuration, a refresh for an image version that does not exist and + an observed-image scan for an image that no longer exists create no run at all. +- A runtime container refresh first synchronizes the configured Docker instances; instances that are enabled in the + database but missing from or disabled in configuration are skipped without a run. +- After every recorded run the inventory gauges are refreshed and the run's duration and status are recorded as + metrics ([Telemetry](telemetry.md)). + +### Overlapping scans + +- Scans of the same observed image never run at the same time within one application process. A second request + for an image that is being scanned waits until the first scan has finished and then runs in full; it is not + skipped or merged. +- Scans of different observed images may run at the same time (for example a manual registration during the + scheduled refresh). +- Runtime container scans and vulnerability refreshes have no such guard: a manual run and a scheduled run of the + same kind may overlap. +- Within the observed-image refresh the images are scanned one after another; within the runtime container refresh + the instances are scanned one after another. Parallelism of vulnerability provider calls inside one refresh is + owned by [Vulnerabilities](vulnerabilities.md). + +### Failure isolation + +- A run of a background job that fails with an error is logged with the job name and its duration and does not stop + the job, the other jobs or the host; the job waits its interval and runs again. +- Inside a batch, an error while scanning one observed image or one Docker instance is logged and the batch goes on + with the next one. +- A failed run is not retried before the next scheduled run. `DockerUpdateGuard:Scanning:RetryCount` (default 2, + range 0–10) applies to individual outbound HTTP requests, not to runs ([Registries](registries.md)). +- Host shutdown cancels the run in progress; it is logged neither as completed nor as failed. + +### Unfinished runs + +A run can stay `Running` when the process stops or crashes during a scan. Such runs are repaired: + +- once when the cleanup job starts, and again at the beginning of every cleanup pass; +- a `Running` run counts as stale once it started longer ago than its threshold, which is twice the configured + interval of its job, but at least 24 hours: + - `ObservedImage`: 2 × `OwnImageBaseScanIntervalMinutes` (the configured floor, not the dynamic interval); + - `RuntimeContainer`: 2 × `RuntimeImageUpdateScanIntervalMinutes`; + - `Vulnerability`: 2 × `VulnerabilityRefreshIntervalMinutes`; +- a stale run is set to `Failed`, gets the repair time as completion time and the error message + "Marked as failed by cleanup: the run never completed (process restart or crash)". + +With the default intervals every threshold is 24 hours, so a run interrupted by a restart shows as `Running` for up +to a day. + +### Retention and cleanup + +Each cleanup pass computes a cutoff of now minus `DockerUpdateGuard:Scanning:RetainScanRunsDays` (default 30, range +1–3650) and removes, in one transaction: + +- **Scan runs:** only completed runs that no update finding, vulnerability finding, container snapshot or image + relationship refers to. Of these, a run is kept only while it is both among the 20 most recently completed and + not completed before the cutoff; all others are removed. Runs still referenced by data are kept regardless of age + and count, and `Running` runs are never removed. +- **Tag candidates:** those whose update finding was resolved before the cutoff, and those without a digest + regardless of age. +- **Update findings:** inactive findings resolved before the cutoff. +- **Vulnerability findings:** inactive findings of image versions that are no longer live, regardless of age, and + inactive findings of live image versions resolved before the cutoff. +- **Container snapshots:** snapshots recorded before the cutoff that have no active update finding and no recorded + container action. +- **Resource samples:** container and Docker instance samples recorded before the cutoff. + +After the removal the cleanup pass refreshes the inventory gauges ([Telemetry](telemetry.md)) and logs the number of +removed items per kind. Nothing else (images, observed images, Docker instances, relationships, tag selections) is +removed by cleanup. + +### Live images + +An image version is *live* when it is: + +- the current version of any observed image (enabled or disabled), or +- running according to the most recent runtime container scan of any Docker instance (for an instance without a + runtime scan, the most recent snapshot of each of its containers counts), or +- a direct base image of one of the above. + +The vulnerability refresh assesses exactly the live image versions, and cleanup uses the same set to purge findings +of image versions that left the fleet. + +### Deployment assumption + +The engine assumes a single application instance per database. Its scan guard is local to one process and the jobs +are not coordinated through the database: several instances against one database duplicate the scheduled work and +may scan the same image at the same time. This is wasteful but does not corrupt the shared image catalog +([Image catalog](image-catalog.md)). + +## Limits and errors + +- Interval, budget and retention keys outside their ranges, or a reserved request share not smaller than the limit + per window, fail configuration validation with a message naming the key and the allowed range + ([Hosting and configuration](hosting.md)). +- The smallest possible interval of every job is one minute; there is no sub-minute schedule and no cron-style + schedule. +- A step that reports a failure or an unexpected error turns the run `Failed`; the error message of the run carries + the step's message as returned by the integration or the error. Integrations report expected failures as results, + not errors ([Registries](registries.md)), so their messages do not contain credentials. +- A run that cannot be saved at its end (for example because the database became unreachable) stays `Running` until + the stale-run repair marks it `Failed`. A vulnerability refresh retries this final save once on a fresh state + before giving up. +- If the database cannot be reached during a cleanup pass, the whole pass fails, nothing is removed, and the next + pass starts after one interval. + +## Related decisions + +None yet. + +## Not here + +- What an observed-image scan resolves: tag metadata and image identity in [Image catalog](image-catalog.md) and + [Registries](registries.md), base images and derived runtimes in [Base images](base-images.md), recommended + updates in [Update detection](update-detection.md). +- Container discovery, snapshots and resource sampling in [Docker instances](docker-instances.md). +- Vulnerability providers, assessment status and the provider parallelism in + [Vulnerabilities](vulnerabilities.md). +- Docker Hub request parallelism, HTTP retry and the failure model of integrations in [Registries](registries.md). +- How scan runs and manual triggers are shown in the UI in [Web UI](web-ui.md). +- Metrics recorded per run and the inventory gauges in [Telemetry](telemetry.md). +- Configuration loading, validation and the startup sequence in [Hosting and configuration](hosting.md). + +## Open questions + +1. The interval of a job is read outside its error handling. If reading it fails — the observed-image refresh counts + enabled images in the database for every interval, and a configuration reload with an invalid value makes the + options unreadable — the job ends with an error, and with the platform's default behavior the host stops. Is a + failing interval lookup meant to stop the host, or should the job fall back to the last known interval? +2. Scan-run retention keeps unreferenced completed runs only while they are both among the 20 most recent and + within `RetainScanRunsDays`, i.e. 20 is a cap, not a minimum kept past retention. Is the cap the intended + behavior (the configuration reference describes `RetainScanRunsDays` as the retention period of the scan + history)? +3. Runtime container scans and vulnerability refreshes can overlap between a manual and a scheduled run, unlike + observed-image scans. Is overlap acceptable for them, or should they get the same per-subject guard? From 650d488118f44d1d843f0d2c75557624761bdde0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:42:18 +0000 Subject: [PATCH 03/12] Describe the hosting and telemetry areas Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/hosting.md | 6 +- docs/areas/telemetry.md | 144 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+), 3 deletions(-) create mode 100644 docs/areas/telemetry.md diff --git a/docs/areas/hosting.md b/docs/areas/hosting.md index f79065e..b520feb 100644 --- a/docs/areas/hosting.md +++ b/docs/areas/hosting.md @@ -12,8 +12,8 @@ every feature area relies on it for validated configuration and a migrated datab ### Configuration sources -- Configuration is read from, in increasing precedence: `appsettings.json`, `appsettings.{Environment}.json`, user - secrets (Development only), environment variables and command-line arguments. Environment variables use `__` as the +- Configuration is read from, in increasing precedence: `appsettings.json`, `appsettings.{Environment}.json`, + environment variables and command-line arguments. Environment variables use `__` as the section separator (`DockerUpdateGuard__Scanning__RetryCount`). - The application reads four top-level sections: `DockerUpdateGuard` (application settings), `Telemetry` (see [Telemetry](telemetry.md)), `ConnectionStrings` and `Logging`. The key-by-key reference with defaults and ranges is @@ -146,7 +146,7 @@ These properties hold today and must be provided for by the deployment: | Invalid `Telemetry` configuration | Startup fails before the host is built (see [Telemetry](telemetry.md)). | | Database unreachable within `MigrationStartupTimeoutSeconds` | The attempts are logged; startup fails. | | Non-transient database error while waiting | Startup fails immediately. | -| Migration fails | Logged; startup fails. A schema is never half-reported as ready. | +| Migration fails | Logged; startup fails. | | Advisory lock cannot be released | Logged; the migration outcome stands. | | Mounted settings file not readable by the container user | Entrypoint exits with code `1` and a fix-it message. | | Certificate import not possible | A message is printed; the container starts anyway. | diff --git a/docs/areas/telemetry.md b/docs/areas/telemetry.md new file mode 100644 index 0000000..678dd30 --- /dev/null +++ b/docs/areas/telemetry.md @@ -0,0 +1,144 @@ +# Telemetry + +## Scope + +What the application emits about itself as logs, traces and metrics, how each signal is switched on and exported, +which application-specific spans and instruments exist and when they change, and what may never appear in them. +Operators rely on it to monitor the application in an OpenTelemetry backend; dashboards and alerts rely on the +names and attributes stated here. + +## Behavior + +### Signals and switches + +- There are three signals: logs, traces and metrics. Each is switched independently by `Telemetry:EnableLogging`, + `Telemetry:EnableTracing` and `Telemetry:EnableMetrics`; all three default to `true`. +- When all three are `false`, no OpenTelemetry pipeline is set up at all and the other `Telemetry` keys are not + validated. +- The platform's console logging is independent of these switches and always active. `Logging:LogLevel` filters apply + to console output and to exported logs alike (shipped defaults: `Information`, and `Warning` for + `Microsoft.AspNetCore`). +- The `Telemetry` section is read once at startup; changing it requires a restart. + +### Export + +- All enabled signals are exported over OTLP to one endpoint, `Telemetry:OtlpEndpoint`. It must be an absolute `http` + or `https` URI. No transport protocol is configured by the application, so the OTLP exporter's default (gRPC, for + example `http://collector:4317`) applies. +- Without `Telemetry:OtlpEndpoint`, enabled signals are collected in-process but nothing is exported. There is no + fallback exporter (no console or Prometheus exporter). +- An unreachable or failing collector affects only the export; scanning, the UI and the database are unaffected. + +### Resource + +Every exported signal carries the same resource: + +| Attribute | Value | +| --- | --- | +| `service.name` | `Telemetry:ServiceName` (default `DockerUpdateGuard`); must not be blank while any signal is enabled. | +| `service.version` | The environment variable `DockerUpdateGuard__DisplayVersion` (the container image sets it); otherwise the application's informational version, file version or assembly version, in that order; otherwise `unknown`. | +| `deployment.environment.name` | `Telemetry:Instance`, only when it is set. The shipped `appsettings.json` sets `Production`; there is no built-in default. | + +### Traces + +- Traces contain a span for every incoming HTTP request and every outgoing HTTP call, plus the application's own + spans from the source `DockerUpdateGuard`. +- Application spans emitted today, all of kind client: + + | Span name | Wraps | Attribute | + | --- | --- | --- | + | `dockerhub.request` | A Docker Hub repository lookup, tag lookup or base-image resolution | `docker.image.reference` (the full image reference) | + | `dockerengine.request` | Container discovery on one Docker instance | `docker.instance.name` (the configured instance name) | + +- The span names `scan.run`, `cve.request`, `portainer.request`, `portainer.action` and `persistence.operation` are + reserved for the application's source but not emitted (see *Open questions*). + +### Logs + +- Exported log records include the category, the formatted message, active scopes and the structured message + parameters as individual attributes. +- Application log events have fixed event IDs and message templates; the startup and database events are + `1000`–`1006` and the inventory refresh is `1010`. Message parameters are named in PascalCase + (`{DockerInstanceName}`, `{ImageReference}`, `{StatusCode}`). + +### Metrics + +- Metrics contain the application's own instruments from the meter `DockerUpdateGuard`, plus the platform's + instruments for incoming HTTP requests, outgoing HTTP calls and the .NET runtime. +- Scan instruments, recorded once per finished scan run after its final status is known (scan runs are defined in + [Scan engine](scan-engine.md)): + + | Name | Type | Unit | Recorded | + | --- | --- | --- | --- | + | `dockerupdateguard.scans.total` | Counter | — | `+1` for every finished scan run | + | `dockerupdateguard.scans.failed` | Counter | — | `+1` for every scan run that finished with status `Failed` | + | `dockerupdateguard.scans.duration` | Histogram | `s` | The run's duration in seconds | + + All three carry `scan.type` (`ObservedImage`, `RuntimeContainer` or `Vulnerability`) and `result.status` (the + final status, for example `Succeeded`, `Partial` or `Failed`). A run that ends `Partial` is counted in + `scans.total` but not in `scans.failed`. +- Inventory gauges: + + | Name | Value | + | --- | --- | + | `dockerupdateguard.observed_images` | Number of observed images, enabled or not | + | `dockerupdateguard.runtime_containers` | Number of distinct containers (per Docker instance and container ID) that have a stored snapshot | + | `dockerupdateguard.base_images.deduplicated` | Number of distinct image versions used as a base image by at least one image | + | `dockerupdateguard.findings.update.active` | Number of active update findings | + | `dockerupdateguard.findings.cve.active` | Number of active vulnerability findings | + | `dockerupdateguard.findings.needs_review.active` | Number of active update findings that are tag recommendations | + +- Gauges report the values of their last refresh; observing them never queries the database. They are refreshed: + - once during startup, before requests are served (see [Hosting](hosting.md)); + - after every finished observed-image scan, runtime-container scan and vulnerability refresh; + - after every cleanup pass. +- Between refreshes a gauge can lag behind the database, for example after a manual registration that triggers no + scan. + +### Credentials and external data + +- Credential values never appear in log messages, log attributes or span attributes: not the Docker Hub PAT, the + Portainer password or API token, the bearer tokens obtained from Docker Hub, registries, Docker Scout or Portainer, + nor the database connection string. Credentials travel only in request headers, and headers are not recorded. +- Database statements are logged without parameter values. +- What logs and spans may carry about external systems: configured instance names, image references, configured base + URLs, the Docker Hub account name, HTTP methods and status codes, counts, durations and error messages from + external tools. Retry log lines carry the request URL without its query string. + +## Limits and errors + +- `Telemetry:OtlpEndpoint` set but not an absolute `http`/`https` URI: startup fails with + `'Telemetry:OtlpEndpoint' must be an absolute http or https URI`. +- `Telemetry:ServiceName` blank while any signal is enabled: startup fails with + `'Telemetry:ServiceName' must be configured when telemetry is enabled`. +- Both checks run while services are registered, before the host is built; a failure aborts the process before the + database is contacted. +- Export failures are handled inside the exporter and are never surfaced to users or scans. + +## Related decisions + +None yet. + +## Not here + +- Deploying the collector and the Grafana/LGTM stack: `INSTALL.md` and `docs/docker-compose.full.yml`. +- What a scan run is and when scans run: [Scan engine](scan-engine.md). +- What counts as a finding, a tag recommendation or a base image: [Update detection](update-detection.md), + [Vulnerabilities](vulnerabilities.md), [Base images](base-images.md). +- Secrets in the UI: [Web UI](web-ui.md); secrets in configuration: [Hosting](hosting.md). +- Runtime warnings for `SkipCertificateValidation` and `AllowInsecureHttp`: [Docker instances](docker-instances.md) + and [Container actions](container-actions.md). + +## Open questions + +1. The span names `scan.run`, `cve.request`, `portainer.request`, `portainer.action` and `persistence.operation` are + defined but no code starts such spans, so scans, vulnerability provider calls, Portainer calls and persistence + have no application span. Should they be emitted, or the names dropped? +2. `dockerupdateguard.runtime_containers` counts every container that still has a stored snapshot, including + containers that have since stopped or been removed, until cleanup deletes their snapshots. Is "containers seen + within the retention period" the intended meaning, or should it count only currently known containers? +3. Configured base URLs (Docker instance, Portainer) are logged verbatim, and the retry log keeps the user-info part of + a URL. A base URL with embedded credentials (`https://user:secret@host`) would therefore reach the logs. Should + URLs be stripped of user info before logging? +4. A scan run that ends `Partial` is not counted in `dockerupdateguard.scans.failed`. Is that the intended alerting + signal, or should partial runs be counted separately? From fb077e738fd8a7f1e263cdec6893c484d9d6b88c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:42:27 +0000 Subject: [PATCH 04/12] Add work-in-progress area documents Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/scan-engine.md | 8 +- docs/areas/update-detection.md | 192 +++++++++++++++++++++++++++++++++ 2 files changed, 196 insertions(+), 4 deletions(-) create mode 100644 docs/areas/update-detection.md diff --git a/docs/areas/scan-engine.md b/docs/areas/scan-engine.md index 7616570..c7217d6 100644 --- a/docs/areas/scan-engine.md +++ b/docs/areas/scan-engine.md @@ -145,8 +145,8 @@ Each cleanup pass computes a cutoff of now minus `DockerUpdateGuard:Scanning:Ret container action. - **Resource samples:** container and Docker instance samples recorded before the cutoff. -After the removal the cleanup pass refreshes the inventory gauges ([Telemetry](telemetry.md)) and logs the number of -removed items per kind. Nothing else (images, observed images, Docker instances, relationships, tag selections) is +After the removal the cleanup pass refreshes the inventory gauges ([Telemetry](telemetry.md)) and logs how many +tag candidates, update findings, vulnerability findings, snapshots and scan runs it removed. Nothing else (images, observed images, Docker instances, relationships, tag selections) is removed by cleanup. ### Live images @@ -176,8 +176,8 @@ may scan the same image at the same time. This is wasteful but does not corrupt - The smallest possible interval of every job is one minute; there is no sub-minute schedule and no cron-style schedule. - A step that reports a failure or an unexpected error turns the run `Failed`; the error message of the run carries - the step's message as returned by the integration or the error. Integrations report expected failures as results, - not errors ([Registries](registries.md)), so their messages do not contain credentials. + the step's message as returned by the integration, or the error's own message. Integrations report expected + failures as results, not errors ([Registries](registries.md)). - A run that cannot be saved at its end (for example because the database became unreachable) stays `Running` until the stale-run repair marks it `Failed`. A vulnerability refresh retries this final save once on a fresh state before giving up. diff --git a/docs/areas/update-detection.md b/docs/areas/update-detection.md new file mode 100644 index 0000000..c5c854f --- /dev/null +++ b/docs/areas/update-detection.md @@ -0,0 +1,192 @@ +# Update detection + +## Scope + +Which update DockerUpdateGuard recommends for a running container image, given the tags its registry publishes: how +tags are parsed and compared, how floating alias tags are resolved to a concrete version, how conservative a major +upgrade is, which assessment status a runtime container gets, and how update findings with their ranked tag candidates +and the manual tag selection per container are kept. Users rely on it to decide what to update; the UI and the +inventory metrics rely on the stored assessments and findings. + +## Behavior + +### Tag formats + +- A **semantic version tag** has the form `[v|V]..[-]` with decimal components, e.g. + `1.27.3`, `v2.0.0-alpine`, `8.0.11-bookworm-slim`. A component too large for a 32-bit integer makes the tag + non-semantic. +- A **version line tag** has the form `[v|V].[-]`, e.g. `8.0` or `1.27-alpine`. It is treated as + an alias of the newest patch of that line, never as a version itself. +- A **year-prefixed tag** has the form `-`, e.g. `2022-CU12-ubuntu-22.04`. +- Every other tag (`latest`, `stable`, `alpine`, ...) is an **alias tag**. +- Tag names are compared case-insensitively throughout. When the registry reports the same tag name twice, only the + most recently published entry counts. + +### Variant families and ordering + +- The suffix of a semantic version tag is split at `-` into segments. A segment whose leading letters are one of + `alpha`, `beta`, `rc`, `pre`, `preview`, `dev`, `snapshot`, `nightly`, `canary`, `milestone`, `ea` (any case) is a + **pre-release segment**; all others are **variant segments**. +- The **variant family** of a semantic tag is its variant segments, each reduced to its leading letters (or kept + whole when it has none), joined by `-`: `1.2.3-alpine3.19` and `1.2.4-alpine3.20` are both family `alpine`; + `1.2.3-rc1` and `1.2.3` are both the plain family. +- Two semantic tags are comparable only within the same variant family. A tag of another family is never a candidate, + so an Alpine tag is never recommended for a Debian one. +- Comparable tags order by major, minor, patch; then by their variant segments in natural order (digit runs compared + numerically, text case-insensitively); then a release ranks above any pre-release of the same version; then + pre-release segments in natural order (`rc2` below `rc10`). +- Year-prefixed tags are comparable only within the same year and the same year-variant family. The year-variant + family is the suffix after its leading servicing segments (segments that start with letters and contain a digit, + such as `CU12`), lower-cased; it is empty when that rest contains no letter. Within a family, tags order by the natural order of the + suffix. + +### Alias resolution + +- An alias or version line tag is resolved to the semantic version tag that carries the same digest. The digest is the + running digest, or, when that is unknown, the registry digest of the alias tag. Without any digest nothing is + resolved. +- For a version line tag only semantic tags of the same major, minor and variant family qualify. +- When several tags qualify, the order of preference is: the variant family of a given reference tag (used when + resolving a recommended alias against the container's current version), then plain tags without a variant, then the + most recently published, then the highest version. +- The resolved tag is stored on each container snapshot as the container's *resolved version*. When it cannot be + resolved in a scan, the resolved version of the most recent earlier snapshot of the same container and the same + image version is carried over, provided the running digest is known. +- When an update is available, the snapshot also stores the *available update version*: the recommended tag itself + when it is a semantic or year-prefixed tag, otherwise the semantic tag the recommended alias resolves to. + +### Input of an evaluation + +- A runtime container is evaluated against at most 250 repository tags requested from its registry for the image's + operating system and architecture ([Registries](registries.md)). +- Before evaluation, tags are dropped that were published before the current tag or that order below the current + version (or, for `latest`, below the version `latest` resolves to), unless they carry the running digest. The + current tag itself is always kept, with the metadata of the direct lookup of the current tag when that succeeded. + +### Evaluation rules + +An evaluation returns one of *up to date*, *update available*, *needs review* or *unknown*, a summary, optional +details, an optional recommended tag and digest, and at most **50** candidates. The first rule that applies wins: + +1. **No tags** — *unknown*, "No registry tag information is available". +2. **`latest` unchanged** — the current tag is `latest` and the running digest equals the registry digest of `latest`: + *up to date*. +3. **Semantic current tag** — candidates are the comparable tags that order above the current tag, are not + pre-releases (unless the current tag is one), do not carry the running digest and were published after the + current tag (when both timestamps are known). They are ordered highest first, ties broken by newest publication, + and filtered by the major-upgrade policy below. If candidates remain, the first is recommended (*update + available*). Otherwise, if the current tag was republished with a different digest, that is the update; otherwise + *up to date*. +4. **Year-prefixed current tag** — candidates are the tags of the same year and year-variant family that order above + the current tag, with the same digest and publication conditions; highest first. No major-upgrade policy applies, + and no other year is ever recommended. Without candidates, a republished current tag is the update; otherwise *up + to date*. +5. **Republished alias** — the registry digest of the current tag differs from the running digest: *update available* + recommending the current tag with its new digest. Its candidates are the tags carrying that new digest: the + current tag first, then tags of the same version line, then semantic tags, then newest and highest. +6. **Version line alias** — the running digest matches a semantic tag of the line: that tag is the current version and + rule 3 is applied to it. The resolved current version is added to the candidates. Without a successor the result is + *up to date*, "The running digest matches version tag '…'". +7. **Anything else** — every other tag that has a digest and was published after the current tag (when both + timestamps are known; newest first, at most 50) is a candidate for *needs review*, "Alternative tags are available and require manual review". With no + such tag the result is *up to date*. + +### Major-upgrade policy + +- When a successor exists in the current major line, the highest of them is recommended. Higher majors are kept as + further candidates but are never recommended. +- When the only successors are in higher majors, only the **next** major (the lowest one above the current) can be + recommended, and only when all of the following hold: + - the next major has at least `DockerUpdateGuard:Scanning:MajorUpgradeMinimumReleaseCount` distinct candidate + versions (default `2`, range `0`–`100`, `0` disables this condition); + - and, unless `DockerUpdateGuard:Scanning:MajorUpgradeMinimumAgeDays` is `0` (default `30`, range `0`–`3650`): + - the earliest known release of the next major (same variant family, with a publication timestamp) is exactly + `.0.0`, so a truncated tag history defers the upgrade; + - that first release was published at least the configured number of days ago; + - the most recent release of the current major line is known, at least that many days old, and not later than the + first release of the next major (the current line is dormant, not maintained next to the new one). +- A recommended major upgrade carries the details "No newer version is available in the `.x` version line". +- A deferred major upgrade yields *up to date*, "No newer version was found in the `.x` version line", with the + details "Tag '…' introduces a new major version and is not recommended yet". A republished current tag (rule 3) + takes precedence over this result. +- The age conditions are measured against the wall clock at evaluation time; a deferred upgrade becomes a + recommendation in a later scan without any other change. + +### Runtime container assessment + +Each container snapshot ([Docker instances](docker-instances.md)) carries an update assessment status and a message +(the summary, followed by the details when present): + +| Status | When | +| --- | --- | +| Not evaluated | Initial value of a new snapshot | +| Up to date | Evaluation result *up to date* | +| Update available | Evaluation result *update available* | +| Manual review required | Evaluation result *needs review* | +| No tag data | Evaluation result *unknown*, or the registry reports the repository as not found | +| Unsupported | No registry client supports the image's registry | +| Failed | The tag lookup failed for any other reason, or processing the container raised an error; the message is the failure message | + +A container that ends *Failed*, *Unsupported* or with *No tag data* because of the registry answer makes its runtime +scan run *partial*, with the message added to the run ([Scan engine](scan-engine.md)). + +### Update findings and tag candidates + +- For *update available* a finding of type *runtime image update* is created, for *needs review* one of type *tag + recommendation*. Each finding references the scan run, the container snapshot, the image version it is about and, + when a tag is recommended, the catalog image version of the recommended tag and digest + ([Image catalog](image-catalog.md)). +- The candidates of the evaluation are stored as tag candidates with tag, digest, publication time, the evaluation + summary as reason, a rank equal to the candidate's 1-based position in the evaluation and a *recommended* flag on the + candidate whose tag equals the recommended tag. Candidates without a digest are not stored, so ranks may have gaps. + A finding holds each tag/digest pair at most once. +- Whenever a container was evaluated (including *no tag data* and *unsupported*), all active findings of the same + container on the same Docker instance are first deactivated: they become inactive, get a resolution timestamp and + lose their tag candidates. When the tag lookup failed, the previous findings stay active. +- After a runtime scan of an instance, the active findings of containers that were no longer discovered are + deactivated the same way. +- Findings of type *base image update* exist in the data model but are not produced. Findings of type *derived base + runtime update* are owned by [Base images](base-images.md). + +### Manual tag selection + +- A user can select one tag, with an optional digest, per runtime container (Docker instance and container id), and + clear it again. A new selection replaces the previous one; tag and digest are stored trimmed, an empty digest as + none, together with the registry repository of the container's latest snapshot and the selection time. +- Clearing a container without a selection does nothing. +- The selection is a record of the user's choice only: it does not change the evaluation, the findings or any + container action. It is shown while the container still runs an image of the same repository it was made for. +- A selection is removed together with its Docker instance or registry repository. + +## Limits and errors + +- At most 250 tags per runtime container evaluation and at most 50 candidates per finding. +- Stored lengths: tag candidate tag 200 characters, digest 255, reason 500; finding summary 500, details 2000; + resolved and available update version 200; assessment message 2000. +- Selecting a tag for a container that has no snapshot fails with "Runtime container '/' + was not found"; nothing is stored. +- Evaluation itself performs no I/O and has no failure state of its own; registry failures reach it only as missing + tags and are reported through the assessment status above, with the registry's failure message. + +## Related decisions + +None yet. + +## Not here + +- Fetching tags, digests and publication times, registry support and retries: [Registries](registries.md). +- Parsing and normalizing image references, the image version catalog: [Image catalog](image-catalog.md). +- Base-image resolution, derived .NET/nginx runtime findings and observed-image findings: [Base images](base-images.md). +- When runtime scans run, scan runs, retention and cleanup of findings and candidates: [Scan engine](scan-engine.md). +- Container discovery and snapshots: [Docker instances](docker-instances.md). +- Applying an update through Portainer: [Container actions](container-actions.md). +- How findings, candidates and the selection are displayed: [Web UI](web-ui.md). + +## Open questions + +1. The manual tag selection is stored and displayed but has no effect on update evaluation or on container actions. + Is it meant to stay informational, or should it pin the recommendation or feed the Portainer update action? +2. Tag candidates without a digest are dropped when a finding is stored, which can leave a *runtime image update* + finding with fewer candidates than the evaluation produced (or none) and ranks with gaps. Is that intended? +3. Findings of type *base image update* are defined but never created. Is the type reserved for future use or + obsolete? From 9525daaa8c603c65c12a8d75bb50ce25fa4d50b5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:43:38 +0000 Subject: [PATCH 05/12] Describe the update detection and vulnerabilities areas Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/update-detection.md | 8 +- docs/areas/vulnerabilities.md | 182 +++++++++++++++++++++++++++++++++ 2 files changed, 186 insertions(+), 4 deletions(-) create mode 100644 docs/areas/vulnerabilities.md diff --git a/docs/areas/update-detection.md b/docs/areas/update-detection.md index c5c854f..2af0303 100644 --- a/docs/areas/update-detection.md +++ b/docs/areas/update-detection.md @@ -37,8 +37,8 @@ inventory metrics rely on the stored assessments and findings. pre-release segments in natural order (`rc2` below `rc10`). - Year-prefixed tags are comparable only within the same year and the same year-variant family. The year-variant family is the suffix after its leading servicing segments (segments that start with letters and contain a digit, - such as `CU12`), lower-cased; it is empty when that rest contains no letter. Within a family, tags order by the natural order of the - suffix. + such as `CU12`), lower-cased; it is empty when that rest contains no letter. Within a family, tags order by the + natural order of the suffix. ### Alias resolution @@ -88,8 +88,8 @@ details, an optional recommended tag and digest, and at most **50** candidates. rule 3 is applied to it. The resolved current version is added to the candidates. Without a successor the result is *up to date*, "The running digest matches version tag '…'". 7. **Anything else** — every other tag that has a digest and was published after the current tag (when both - timestamps are known; newest first, at most 50) is a candidate for *needs review*, "Alternative tags are available and require manual review". With no - such tag the result is *up to date*. + timestamps are known; newest first, at most 50) is a candidate for *needs review*, "Alternative tags are + available and require manual review". With no such tag the result is *up to date*. ### Major-upgrade policy diff --git a/docs/areas/vulnerabilities.md b/docs/areas/vulnerabilities.md new file mode 100644 index 0000000..fb6548a --- /dev/null +++ b/docs/areas/vulnerabilities.md @@ -0,0 +1,182 @@ +# Vulnerabilities + +## Scope + +How DockerUpdateGuard enriches image versions with known vulnerabilities: the choice of provider (Trivy, Docker Scout +or none), how each provider is called and its answer mapped, which image versions a refresh covers, the vulnerability +assessment per image version, the lifecycle of vulnerability findings, the manual rescan and the configuration hints. +Users rely on it to see which running or observed images carry which CVEs; operators rely on it to configure a +provider; the UI and the inventory metrics rely on the stored assessments and findings. + +## Behavior + +### Configuration + +All keys live under `DockerUpdateGuard:Vulnerabilities`: + +| Key | Default | Rule | +| --- | --- | --- | +| `Enabled` | `false` | Switches vulnerability refresh on | +| `Provider` | `None` | `None`, `DockerScout` or `Trivy`; must not be `None` when `Enabled` is `true` | +| `TrivyBaseUrl` | none | Absolute `http`/`https` URI; required when `Enabled` is `true` and `Provider` is `Trivy` | +| `TrivyExecutablePath` | `trivy` | Executable to start; resolved through `PATH`; an empty value means `trivy` | +| `DockerScoutLoginUrl` | `https://hub.docker.com/v2/users/login` | Absolute `http`/`https` URI, always validated | +| `DockerScoutBaseUrl` | `https://api.scout.docker.com` | Absolute `http`/`https` URI, always validated | +| `RequestTimeoutSeconds` | `120` | `1`–`300` | +| `MaxParallelScans` | `1` | `1`–`8` | + +- A configuration that breaks one of these rules fails startup ([Hosting](hosting.md)). +- The provider is chosen anew at the start of every refresh from the current configuration, so a reloaded + configuration takes effect with the next refresh. When `Enabled` is `false`, a refresh does nothing: no provider is + called, no scan run is recorded and stored assessments and findings stay as they are. +- Docker Scout uses the Docker Hub credentials `DockerUpdateGuard:DockerHub:UserName` and + `DockerUpdateGuard:DockerHub:Pat`; they are not required by validation. + +### What a refresh covers + +- A full refresh scans every **live** image version: the current image version of every observed image, the image + versions of the most recent runtime container scan of each Docker instance (for an instance without such a scan, + the latest snapshot of each container), and the recorded base images of all of these + ([Base images](base-images.md)). +- A full refresh runs on the schedule `DockerUpdateGuard:Scanning:VulnerabilityRefreshIntervalMinutes` (default `180`, + range `1`–`10080`, see [Scan engine](scan-engine.md)) and can be started manually from the scan history page. +- A **manual rescan** of a single image version can be started from the observed image, own image and runtime + container lists. It follows the same rules for that one image version. An image version that does not exist is + ignored without a scan run. +- Every refresh is recorded as one scan run of type *vulnerability* with trigger *scheduled* or *manual*, also when it + covers no image version. +- Provider calls run in batches of `MaxParallelScans` image versions at a time; the results of a batch are applied and + saved one image version after the other. + +### Assessment per image version + +Every scanned image version gets an assessment status, the provider it came from, a message, the check time and the +scan run: + +| Status | When | Message | +| --- | --- | --- | +| Not scanned | No assessment stored yet | — | +| No findings | Provider succeeded with no advisory | "The vulnerability provider reported no findings" | +| Findings detected | Provider succeeded with advisories | "The vulnerability provider reported N finding(s)", N after deduplication | +| Not configured | Provider not configured (missing Trivy URL or Docker Hub credentials) | Provider message | +| Unsupported | Provider cannot scan this image (Docker Scout and a registry other than `docker.io`) | Provider message | +| Unavailable | Image or tag not found by the provider | Provider message | +| Failed | Any other provider failure | Provider message | + +- A failed assessment leaves the image version's findings untouched. +- A failed assessment adds its message to the scan run. The run ends *failed* once any provider call returned + *failed* (or the run itself failed), otherwise *partial* when any assessment was not successful, otherwise + *succeeded*. + +### Findings + +- A finding is identified within an image version by advisory id and affected package, compared case-insensitively. + When a provider reports the same pair more than once, the entry with the highest severity, then the highest CVSS + score, is kept. +- For each reported advisory: a matching active finding is updated; a matching inactive finding (the most recently + resolved one) is reactivated with a new detection time and the current scan run, and updated; otherwise a new + finding is created. Updating overwrites title, severity, source, installed and fixed version, fix status, package + class and type, CVSS score, summary and reference URL. +- A finding that was active before a successful scan of its image version and is not reported again is deactivated + with a resolution time and the resolving scan run. +- At the end of a full refresh, every active finding of an image version that is neither live nor scanned in this run + is deactivated the same way. A manual rescan never does this. +- At most one active finding exists per image version, advisory id and affected package. +- Severity is one of *not set*, *low*, *medium*, *high*, *critical*. The source is the configured provider (*Trivy* or + *Docker Scout*). Fix status is one of *not set*, *fixed*, *affected*, *will not fix*, *fix deferred*, *end of life*. + +### Trivy + +- Trivy runs as a local process in client mode against the Trivy server at `TrivyBaseUrl`; the image is analysed by + the process, only the vulnerability database lookup goes to the server. No shell is involved; each argument is + passed separately: + `image --server --format json --quiet --scanners vuln --pkg-types os,library + --timeout s `. +- The target is `:` for `docker.io` images and `/:` otherwise, followed by + `@` when the digest is known. +- The process gets `RequestTimeoutSeconds` plus 10 seconds. When that passes, the process and its children are killed + and the result is *failed*, "Trivy scan timed out after N seconds". +- A non-zero exit code whose error output contains `MANIFEST_UNKNOWN`, `manifest unknown`, or both `manifest` and + `not found` (any case) means the image is gone from the registry: *unavailable*, "The image is no longer available + in the registry. The tag was likely re-published with a newer digest, leaving this version orphaned." Any other + non-zero exit code is *failed*, "Trivy exited with code X: ". +- The JSON report on standard output is mapped per vulnerability of every result block: advisory id from + `VulnerabilityID`; title from `Title`, else the advisory id; severity from `Severity` (`LOW`, `MEDIUM`, `HIGH`, + `CRITICAL`, any case, everything else *not set*); affected package, installed and fixed version from `PkgName`, + `InstalledVersion`, `FixedVersion`; fix status from `Status` (`fixed`, `affected`, `will_not_fix`, `fix_deferred`, + `end_of_life`, any case, everything else *not set*); package class and type from the result block's `Class` and + `Type`; summary from `Description`; reference URL from `PrimaryURL`, else the first of `References`. +- The CVSS score is the NVD v3 score, else the first v3 score of any source, else the NVD v2 score, else the first v2 + score of any source, else none. + +### Docker Scout + +- Docker Scout scans only `docker.io` images, by tag; the digest is not used. A repository without a namespace is + looked up under `library`. +- The provider logs in by posting the Docker Hub user name and token to `DockerScoutLoginUrl` and caches the returned + token for the whole process until one minute before the expiry stated in the token, or for five minutes when the + expiry cannot be read. Only one login runs at a time. A failed login gives *failed*, "Docker Hub authentication + failed". +- Vulnerabilities are read from `/v1/repositories///tags//vulnerabilities`, + following the `next` link of each page, for at most **50 pages**. When the cap is reached, the advisories read so + far are returned as a successful result. +- HTTP 401 discards the cached token, logs in once more and reads again from the first page; a second 401 gives + *failed*, "Docker Scout rejected the authentication token". HTTP 404 gives *unavailable*, "Image not + found in Docker Scout". Any other unsuccessful status gives *failed*, "Docker Scout request failed with status X". +- Each HTTP request is limited to `RequestTimeoutSeconds`. +- Mapping: advisory id and title from `id`, severity from `severity` (as for Trivy), summary from `description`, + reference URL from the first of `links`. Docker Scout supplies no package, versions, fix status or CVSS score. + +### Configuration hints + +The dashboard shows at most one hint about the vulnerability configuration: + +- *Scanning disabled* when `Enabled` is `false`. +- *Trivy base URL missing* when `Provider` is `Trivy` and the most recent completed vulnerability scan run (by start + time) ended *partial* or *failed* with a message containing "TrivyBaseUrl is not configured". +- *Docker Hub credentials missing* when `Provider` is `DockerScout` and that run ended *partial* or *failed* with a + message containing "Docker Hub credentials are required for Docker Scout". +- No hint otherwise. + +## Limits and errors + +- Providers never raise errors for expected failures; they report *not configured*, *unsupported*, *not found* or + *failed* with a message ([Registries](registries.md)). An unexpected error inside a provider call becomes *failed* + with "Trivy scan failed: …" or "Docker Scout scan failed: …". Cancellation is not converted. +- An unexpected error during a refresh ends the scan run *failed* with the error message; work saved before stays + saved. If saving the final run state fails, it is retried once. +- Stored lengths: advisory id 100 characters, title 300, affected package 200, installed and fixed version 100 each, + package class and type 50 each, CVSS score 5 digits with 2 decimals, assessment message 2000; summary and reference + URL are unlimited. +- Messages are built from the image reference and the provider's error text; no credentials or tokens are put into + them. + +## Related decisions + +None yet. + +## Not here + +- Refresh schedule, scan run statuses, stuck-run repair and cleanup of resolved findings: [Scan engine](scan-engine.md). +- Which base images an image version has: [Base images](base-images.md). +- Docker Hub credentials, registry access and the retry of transient HTTP failures: [Registries](registries.md). +- Validation at startup, the bundled Trivy CLI in the container image: [Hosting](hosting.md). +- How assessments, findings, severities and the rescan control are displayed: [Web UI](web-ui.md). +- Inventory gauges for active findings: [Telemetry](telemetry.md). + +## Open questions + +1. Docker Scout follows the `next` link of a response as given and sends the bearer token with it, without checking + that it points to `DockerScoutBaseUrl`. Should the link be restricted to the configured host? +2. Docker Scout requests do not go through the transient-failure retry that the other outbound HTTP calls use. Is that + intended? +3. Docker Scout scans the tag, not the digest, so an image version whose tag was republished is assessed with the + advisories of the newer image. Is that acceptable? +4. Provider values longer than the stored lengths (for example a Trivy title over 300 characters) are not shortened; + saving them fails and ends the whole refresh *failed*, leaving the remaining image versions unscanned. Should they + be truncated? +5. Validation rejects `Enabled` with `Provider` `Trivy` and no `TrivyBaseUrl`, so the *Trivy base URL missing* hint + and the Trivy *not configured* result look unreachable (unless an invalid reloaded configuration is still served). + Are they kept on purpose? +6. The finding sources *registry advisory* and *manual* exist in the data model but are never produced. Reserved or + obsolete? From ef66fb6bfd46143c55522c43085e4dd21e976a82 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:43:43 +0000 Subject: [PATCH 06/12] Describe the scan engine and web UI areas Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/scan-engine.md | 8 +- docs/areas/web-ui.md | 162 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+), 4 deletions(-) create mode 100644 docs/areas/web-ui.md diff --git a/docs/areas/scan-engine.md b/docs/areas/scan-engine.md index c7217d6..08ff394 100644 --- a/docs/areas/scan-engine.md +++ b/docs/areas/scan-engine.md @@ -50,8 +50,8 @@ interval = max(OwnImageBaseScanIntervalMinutes, ⌈DockerHubRequestLimitWindowHo - With no enabled observed image the interval is `OwnImageBaseScanIntervalMinutes`. - The count of enabled observed images is read again before every wait. - Defaults and ranges: `DockerHubRequestLimitWindowHours` 6 (1–168), `DockerHubRequestLimitPerWindow` 200 - (1–100000), `DockerHubReservedManualRequestsPerWindow` 40 (0–100000). The reserved share must be smaller than the - limit per window, so the budget is always at least 1. + (1–100000), `DockerHubReservedManualRequestsPerWindow` 40 (0–100000). The reserved share must be smaller than + the limit per window, so the budget is always at least 1. - With the defaults, up to 26 enabled images are scanned every 60 minutes; from 27 images on the interval grows by about 2.25 minutes per image (e.g. 100 images → 225 minutes). - The formula only spaces out scheduled runs. Manual scans are not counted against the budget, and the job does not @@ -146,8 +146,8 @@ Each cleanup pass computes a cutoff of now minus `DockerUpdateGuard:Scanning:Ret - **Resource samples:** container and Docker instance samples recorded before the cutoff. After the removal the cleanup pass refreshes the inventory gauges ([Telemetry](telemetry.md)) and logs how many -tag candidates, update findings, vulnerability findings, snapshots and scan runs it removed. Nothing else (images, observed images, Docker instances, relationships, tag selections) is -removed by cleanup. +tag candidates, update findings, vulnerability findings, snapshots and scan runs it removed. Nothing else (images, +observed images, Docker instances, relationships, tag selections) is removed by cleanup. ### Live images diff --git a/docs/areas/web-ui.md b/docs/areas/web-ui.md new file mode 100644 index 0000000..b27c356 --- /dev/null +++ b/docs/areas/web-ui.md @@ -0,0 +1,162 @@ +# Web UI + +## Scope + +The browser interface of DockerUpdateGuard: its pages and routes, what each page shows, which actions a user can +start from it, how open pages pick up changes, the error and not-found pages, and light/dark mode. Users rely on it +to read the state the other areas compute and to start manual scans; it computes no findings of its own. + +## Behavior + +### General + +- The UI is a server-rendered web application. Every page is rendered on the server first and then stays + interactive over a persistent connection to the server; there is no client-side application logic beyond that + connection. Data loaded for the first render is reused when the page becomes interactive, not loaded twice. +- There is no sign-in. Everyone who can reach the application can see every page and start every action + ([Hosting and configuration](hosting.md) owns the security posture). +- All pages share one layout: a navigation drawer, a top bar and the page content. The drawer collapses into a menu + button on narrow screens. +- The top bar shows the product name, the dark-mode toggle, the number of *protected assets* (manual observed images + plus own images plus runtime containers) and *last activity* (the start time of the most recent scan run, or "No + scans yet"). The top bar reloads these values on every navigation and on every dashboard refresh signal (see + *Refresh without reload*). +- The navigation lists Dashboard, My Images, Observed Images, Runtime Containers, Docker Instances, Base Images, + Vulnerabilities and Scan History. *My Images* is listed only when both `DockerUpdateGuard:DockerHub:UserName` and + `DockerUpdateGuard:DockerHub:Pat` are set; *Base Images* only when at least one base image has been resolved. Both + conditions are evaluated when the session starts. The pages stay reachable by their routes either way. +- The drawer shows the application version: `DockerUpdateGuard:DisplayVersion` when set, otherwise the version of the + running build, otherwise "unknown". +- Timestamps are shown as short date and short time in the time zone and culture of the server process, not of the + browser. +- While a page loads its data it shows placeholder skeletons; a list without data shows an explanatory empty state. + +### Pages and routes + +| Route | Page | Shows | +| --- | --- | --- | +| `/` | Dashboard | Counts of my images, manual observed images, Docker instances, runtime containers and resolved base images; active update findings; active derived-runtime warnings of own images; active vulnerability findings of currently running containers with a per-severity breakdown; a configuration hint for vulnerability scanning; the 10 most recent scan runs | +| `/my-images` | My Images | Images discovered from the configured Docker Hub account, with latest scan status and message, number of running containers, derived-runtime hint and vulnerability status | +| `/my-images/{id}` | My Image details | One observed image: reference, latest scan, derived-runtime hint, running containers, base image chain, update findings with tag candidates, and — when there are any — vulnerability findings and its 20 most recent scan runs | +| `/observed-images` | Observed Images | The registration form and the manually registered images with latest scan, active update findings, derived-runtime hint and vulnerability status | +| `/observed-images/{id}` | Observed Image detail | One observed image: reference, latest scan, whether it is an own or a manual image, derived-runtime hint, running containers (own images only), base image chain, vulnerability findings and update findings with tag candidates; no scan history | +| `/runtime-containers` | Runtime Containers | Every currently known container with instance, tag (and the version an alias resolves to), CPU, memory and network usage with a short trend of up to the 12 latest samples, update state, available update and vulnerability status; a link to the own image when the container runs one | +| `/runtime-containers/{instanceId}/{containerId}` | Runtime Container detail | One container: image and digest, Compose project, linked own image, update state and message, derived-runtime hint, base image chain, resource usage history, manual tag preference, available tag candidates, update findings, vulnerability findings and the scans of its 20 most recent snapshots | +| `/docker-instances` | Docker Instances | Configured Docker instances with endpoint, connection kind, latest runtime scan status and time, current resource usage and number of containers; with exactly one instance its current usage and usage history are shown on this page directly | +| `/docker-instances/{id}` | Docker Instance detail | The containers currently observed on one instance with linked own image and resource usage | +| `/base-images` | Base Images | Resolved base images with the images built on them, the runtime containers and number of observed images that use them, and their vulnerability status | +| `/vulnerabilities` | Vulnerabilities | Active advisories across all images, one row per advisory, with severity and CVSS score, affected packages, fixed versions, number of affected images and running containers; a row expands to the affected images | +| `/scan-history` | Scan History | The 20 most recent scan runs with type, status, trigger, target, start time and error message | +| `/error` | Error | "Unexpected error" — the request could not be completed | +| `/not-found` | Not found | "Page not found" with a link back to the dashboard | + +- Detail routes accept only identifiers in GUID format (the container identifier of the runtime container route is + free text); any other path renders the not-found page. +- Resource usage histories on detail pages show up to the 48 most recent samples of the last 24 hours. +- The Vulnerabilities page filters by free text (advisory identifier, title or package, case-insensitive) and by + severity (All, Critical, High, Medium, Low), shows 25 rows per page with page sizes 25, 50 and 100, and links an + advisory to its reference only when the reference is an absolute `http` or `https` address; the link opens in a + new tab. +- Which values the pages show and how they are computed (update state, recommended tag, vulnerability status, + derived runtime, base image chain, resource values) is owned by the feature areas listed under *Not here*. + +### Actions + +| Action | Where | Effect | +| --- | --- | --- | +| Register and scan | Observed Images | Registers an image from name, image reference and optional description ([Image catalog](image-catalog.md)), scans it immediately as a manual observed-image scan and opens its detail page | +| Check image updates | Runtime Containers | Runs a manual runtime container scan of all enabled Docker instances | +| Refresh vulnerabilities | Scan History, shown only when vulnerability scanning is enabled | Runs a manual vulnerability refresh of all live images; the page also shows when the next scheduled refresh is expected | +| Rescan (per image) | Observed Images, My Images, Runtime Containers | Runs a manual vulnerability refresh of that one image version; next to it the time of the last check or "Never scanned" | +| Select / Clear selection | Runtime Container detail | Stores or removes a manual tag preference for that container ([Update detection](update-detection.md)); it does not change or redeploy the container | + +- Every action runs while the user waits: its button is disabled and shows progress until the action has finished, + and the page then reloads its data. Starting the same action twice from one page is not possible while it runs. +- Manual scans are recorded as scan runs with trigger `Manual` ([Scan engine](scan-engine.md)). +- The per-image rescan is offered even when vulnerability scanning is disabled; it then completes without doing + anything and records no run. +- Name and image reference are required for a registration; the form reports a missing value before anything is + sent. +- The expected time of the next scheduled vulnerability refresh is an estimate: the start of the most recent + vulnerability run of any trigger plus `VulnerabilityRefreshIntervalMinutes`, or "shortly (no run recorded yet)". +- The UI offers no container actions (restart, redeploy, update) and no way to edit configuration, delete images or + enable and disable them. + +### Refresh without reload + +- After a manual scan, registration or rescan finishes, every open dashboard and every open top bar in every browser + session connected to the same application process reloads its data without a page reload. +- Scheduled background runs do not send this signal: their results become visible when a page is opened or + navigated to again (the top bar also refreshes on every navigation). No page polls the server. + +### Vulnerability configuration hint + +The dashboard shows at most one hint about vulnerability scanning: + +- scanning disabled: how to enable it (`DockerUpdateGuard:Vulnerabilities:Enabled` and a provider); +- Trivy not configured: set `DockerUpdateGuard:Vulnerabilities:TrivyBaseUrl`; +- Docker Scout not configured: set `DockerUpdateGuard:DockerHub:UserName` and `DockerUpdateGuard:DockerHub:Pat`. + +When the hint applies is owned by [Vulnerabilities](vulnerabilities.md). The hint names configuration keys only, +never their values. + +### Light and dark mode + +- The UI has a light and a dark theme, switched with the toggle in the top bar. +- The choice is remembered per browser in encrypted browser storage. Without a stored choice the theme follows the + operating system's color-scheme preference, determined after the first render; the server-rendered first paint + is always light. +- If the browser storage cannot be read or written, the toggle still works for the session and the choice is not + remembered. +- Users who ask their system for reduced motion get (practically) no transitions or animations. + +### Secrets + +- The UI never shows configured secrets: the Docker Hub token, Portainer credentials and token and the database + connection string are neither displayed nor sent to the browser. Pages show only whether an integration is + configured, and the Docker Hub user name as the title of the My Images list. +- Docker instance endpoints are shown as configured. + +## Limits and errors + +- An unhandled error during a request outside the Development environment renders the `/error` page; an unknown + route or an HTTP error status renders the `/not-found` page. The HTTP-level handling is owned by + [Hosting and configuration](hosting.md). +- An error in a user action is shown as an error message on the page that started it (for the per-image rescan, next + to its button); the page stays usable and the action can be started again. The message is the error's own message. +- Error messages of scan runs are shown as stored ([Scan engine](scan-engine.md)) in the dashboard, the scan history, + the latest-scan columns and the detail pages. +- If the connection to the server is lost, the page stops being interactive until the browser reconnects or the page + is reloaded. +- Lists are not paged except the Vulnerabilities page; all other lists show every item or the fixed number stated + above. + +## Related decisions + +None yet. + +## Not here + +- How update states, recommended tags, tag candidates and manual tag preferences are evaluated in + [Update detection](update-detection.md). +- Image registration rules and reference formats in [Image catalog](image-catalog.md). +- Base image chains, shared base images and derived runtime hints in [Base images](base-images.md). +- Vulnerability status, severities, fix status and the configuration hint rules in + [Vulnerabilities](vulnerabilities.md). +- Container discovery and resource values in [Docker instances](docker-instances.md). +- Scan runs, schedules and retention in [Scan engine](scan-engine.md). +- Container actions through Portainer in [Container actions](container-actions.md). +- Error and status-code middleware, HTTPS and the missing authentication in + [Hosting and configuration](hosting.md). + +## Open questions + +1. The scan status chips (dashboard, scan history, observed image and My Images lists) color `Failed`, `Running` and + `Pending`, but expect `Completed` for success, a status that does not exist; `Succeeded` and `Partial` are shown + without color. Should `Succeeded` be green and `Partial` be marked as a warning? +2. A detail route with a well-formed identifier that does not exist (for example a deleted observed image) keeps + showing the loading skeleton instead of a "not found" state. Is that intended? +3. Docker instance endpoints are shown exactly as configured. Should an endpoint that embeds credentials in its + address (user information in the URL) be masked, or is such configuration unsupported? +4. The Scan History and dashboard descriptions mention discovery and cleanup runs, but those jobs record no scan + runs. Should the text change, or should these jobs record runs? From 5ba90c1a7827f4f85ee82cecf0adf5e12e50f730 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:45:57 +0000 Subject: [PATCH 07/12] Describe the registries, image catalog and base images areas Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/base-images.md | 147 ++++++++++++++++++++++++++ docs/areas/image-catalog.md | 119 +++++++++++++++++++++ docs/areas/registries.md | 204 ++++++++++++++++++++++++++++++++++++ 3 files changed, 470 insertions(+) create mode 100644 docs/areas/base-images.md create mode 100644 docs/areas/image-catalog.md create mode 100644 docs/areas/registries.md diff --git a/docs/areas/base-images.md b/docs/areas/base-images.md new file mode 100644 index 0000000..fc6d55f --- /dev/null +++ b/docs/areas/base-images.md @@ -0,0 +1,147 @@ +# Base images + +## Scope + +Which images an observed image is built on, and what follows from that: the observed-image scan that resolves the +base-image chain from image labels, the base-image relationship graph it stores, the shared base images view that shows +which observed images and running containers depend on the same base, and the derived runtime detection that recognizes +a .NET or nginx runtime inside an image and compares it with the upstream release feeds. Users rely on it to see where +a base-image update or vulnerability reaches; vulnerability refreshes and the dashboard rely on the stored graph. + +## Behavior + +### Observed-image scan + +- An observed-image scan works on the observed image's current image version ([Image catalog](image-catalog.md)) in + three steps, each reading registry data as described in [Registries](registries.md): + 1. **Tag refresh** — a tag lookup for the current version. On success its publication time and tag metadata are + stored on the version. Otherwise the scan becomes *failed* (outcome *failed*) or *partial* (any other outcome). + 2. **Base-image resolution** — described below. + 3. **Derived runtime check** — described below. +- The scan ends *succeeded* when no step reported a problem, *partial* or *failed* as the steps decide (*failed* is + never lowered to *partial*). An unexpected error in any step makes the scan *failed*. The messages of all steps are + stored on the scan run, each distinct message once. When scans run, overlap rules and scan-run bookkeeping: + [Scan engine](scan-engine.md). + +### Resolving the chain + +- Base images are resolved only for observed images. Running containers are not resolved themselves; they share an + observed image's chain only when they run exactly that image version. +- The source is the image configuration's labels: `org.opencontainers.image.base.name` names the base image (parsed as + an image reference, see [Image catalog](image-catalog.md)); `org.opencontainers.image.base.digest` supplies its + digest (lowercased) when the name carries none. The same rule applies to Docker Hub and to every other registry. +- The chain is followed by reading each base image's own configuration and labels. Depth 1 is the direct base; the + chain ends at depth 5, at an image without the label, or at a base whose configuration cannot be read (the chain + found so far is kept). +- An image without the label has an empty chain; that is a successful resolution. +- When the observed image's own configuration cannot be read, resolution is unavailable: the stored chain is left + unchanged and the scan becomes *failed* (outcome *failed*) or *partial* (any other outcome). +- Only base-image relationships are produced. The relationship type for build stages exists in storage but nothing + records it. + +### Storing the chain + +- A successful resolution replaces the stored base-image relationships of the current version: the previous ones are + removed and the new chain (possibly empty) is recorded. Relationships belong to the image version, so those of an + earlier current version remain when the observed image moves to a new version. +- For each base image a tag lookup refreshes the exact base. Its stored tag is: + - the label's tag when it is a specific version tag (semantic or year-prefixed, see + [Update detection](update-detection.md)) or when no digest is known; + - otherwise (a floating tag such as `latest` with a digest) the specific version tag of the same repository that + points at the same digest, searched among at most 150 tags — restricted to the floating tag's version line unless + it is `latest`, and to tags published since the floating tag itself; when none is found, the label's tag. +- The base image's catalog entry is the stored tag plus the digest. When the tag lookup succeeded, its publication + time and tag metadata are stored on that entry; when it did not, the relationship is still recorded and the scan + becomes *partial* (unless already *failed*) with the lookup's message. +- A relationship records the child (the observed image's current version), the base, the depth, the source reference + (the image whose label named this base: the observed image for depth 1, the previous base for deeper levels) and the + scan run that recorded it (cleared when that run is deleted). +- A relationship is unique per child, base, depth and type; depth is greater than zero and child and base differ. + +### Shared base images + +- `/base-images` lists every base image recorded for the current version of any observed image (enabled or not), at + every depth. An entry used by a single observed image is listed too. +- Entries are grouped case-insensitively: a base with a digest by `registry/repository@digest` (tags that resolve to + the same digest merge); a base without a digest by `registry/repository:tag` together with its source reference. +- Each entry shows the number of distinct observed images using it, the references of the images that name it, the + running containers whose latest snapshot is exactly one of those image versions, and a vulnerability summary of the + base ([Vulnerabilities](vulnerabilities.md)). +- Entries are ordered by number of running containers, then number of observed images (both descending), then by + reference. + +### Derived runtime detection + +- Recognizes the .NET or nginx runtime an image is built on, also without base-image labels. One runtime per image. +- Inputs: for observed images the environment variables of the image configuration from the registry; for running + containers the environment of the local image and its build history (each entry's command and comment) from the + Docker Engine ([Docker instances](docker-instances.md)). +- Signals, strongest first; on equal strength environment variables win over history, and earlier history entries + over later ones: + + | Strength | Signal | + | -------- | ------ | + | 0 | Environment variable `DOTNET_VERSION` | + | 1 | Environment variable `NGINX_VERSION`; `DOTNET_VERSION` in history | + | 2 | Environment variable `ASPNET_VERSION`; `NGINX_VERSION` in history | + | 3 | `ASPNET_VERSION` in history | + | 4 | A history reference to `mcr.microsoft.com/dotnet/aspnet`, `runtime`, `runtime-deps` or `sdk` with tag `x.y.z` | + +- Variable names are matched case-insensitively; in history the name must be followed by `=`, `:` or whitespace and a + version. The version is the first `major.minor.patch` in the value (anything after it is ignored); a value without + three numeric parts is no signal. The channel is `major.minor`. + +### Release feeds and runtime findings + +- **.NET:** `releases-index.json` under `DockerUpdateGuard:ReleaseMetadata:DotNetBaseUrl` (default + `https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/`). Per channel (`channel-version`) the latest + runtime (`latest-runtime`) and whether that release is a security release (`security`). +- **nginx:** the `download/` page under `DockerUpdateGuard:ReleaseMetadata:NginxBaseUrl` (default + `https://nginx.org/`). Every `nginx-x.y.z.tar.gz` link counts; a channel's latest is its highest version. +- Both URLs are fixed at startup and must be absolute `http`/`https` URLs ([Hosting](hosting.md)). A feed is loaded at + most once per scan and not kept between scans. A channel missing from the feed gives no finding. +- A runtime is outdated when the feed's latest version of the same channel is higher than the detected one. Only + patch updates within the channel are reported; a newer channel is never suggested. +- An outdated runtime yields an update finding of type *derived base runtime update*: + - on an observed image: summary "Own image uses an outdated .NET base runtime" (discovered image) or "Observed image + uses an outdated .NET base runtime" (manual image), likewise for NGINX. The details name the detected and the + latest version, add for .NET that the latest release includes security fixes when it does, and advise to rebuild + and republish (own image) or that the upstream publisher must rebuild (manual image); + - on a running container: the same check, made only when the container's update evaluation completed + ([Update detection](update-detection.md)); the summary says "Own image …" when the container runs a discovered + image's repository, otherwise "A linked image uses an outdated … base runtime". +- On an observed image the previous findings are replaced: with no runtime detected they are removed; with a runtime + detected and the feed available they are removed and a new one is added if still outdated; when the configuration + or the feed cannot be read they stay as they are. + +## Limits and errors + +- Chain depth is capped at 5; base-tag resolution searches at most 150 tags. +- Registry outcomes other than *succeeded* never abort the scan; they set *partial* or *failed* and add the registry's + message (status code and response body, see [Registries](registries.md)). +- Feed failures (non-success status, missing `releases-index` array, unreadable content) only suppress the runtime + finding; they do not change the scan status. Unreadable content is logged as a warning. + +## Related decisions + +None yet. + +## Not here + +- Reading configurations, tags and labels from registries, retry and timeouts: [Registries](registries.md). +- Parsing references and the catalog entries: [Image catalog](image-catalog.md). +- Tag formats, version lines and update recommendations: [Update detection](update-detection.md). +- Vulnerabilities of base images: [Vulnerabilities](vulnerabilities.md). +- Schedules, overlap rules and scan-run retention: [Scan engine](scan-engine.md). +- Page layout of `/base-images` and the detail pages: [Web UI](web-ui.md). + +## Open questions + +1. Running containers are only connected to a base-image chain when they run the very image version an observed image + points at (same tag and digest); a container on a different digest of the same tag shows no base images. Should + containers be resolved, or matched by repository? +2. The build-stage relationship type is defined but never produced. Should it be removed or implemented? +3. The shared base images view lists bases used by a single observed image and counts disabled observed images. Is + "shared" meant to require at least two users, and should disabled images count? +4. Derived runtime detection for observed images uses environment variables only, without build history, so the + history-based signals apply only to running containers. Intended? diff --git a/docs/areas/image-catalog.md b/docs/areas/image-catalog.md new file mode 100644 index 0000000..7599398 --- /dev/null +++ b/docs/areas/image-catalog.md @@ -0,0 +1,119 @@ +# Image catalog + +## Scope + +What "an image" is in DockerUpdateGuard: how image references are parsed and normalized, the deduplicated catalog of +registry repositories and image versions (repository, tag and digest) that every other area points at, how the catalog +stays free of duplicates under concurrent writers, and observed images — the images a user registers or the Docker Hub +account discovery adds to be tracked, with their source and enabled state. Every scanning area and the UI rely on it to +refer to the same image the same way. + +## Behavior + +### Image references + +- A reference has the form `[registry/]repository[:tag][@digest]`; surrounding whitespace is ignored. +- **Digest:** everything after the first `@`, trimmed and lowercased. +- **Tag:** the text after the last `:` that follows the last `/`, trimmed, with its case kept. A missing or empty tag + is `latest`, also for a reference that names only a digest. +- **Registry:** the first path segment, lowercased, when the reference has at least two segments and that segment + contains a `.` or a `:` or equals `localhost` (case-insensitive). Otherwise the registry is `docker.io`. +- **Repository:** the remaining segments joined with `/`, lowercased; empty segments are ignored. A reference with a + single segment gets the `library/` namespace (`nginx` → `docker.io/library/nginx:latest`). +- **Mirror-wrapped references:** when the registry is `docker.io` and the first repository segment is + `mcr.microsoft.com`, `ghcr.io`, `quay.io` or a host-like name containing `harbor`, that segment becomes the registry + (`docker.io/ghcr.io/owner/app:1` → `ghcr.io/owner/app:1`). +- No other validation happens when parsing: characters and lengths are not checked. A reference without a repository + name is rejected. +- Docker Hub spellings are not unified: `docker.io/nginx` has the repository `nginx`, not `library/nginx`, and + `index.docker.io/…` is a different registry from `docker.io/…`. Registry access still treats them alike + ([Registries](registries.md)), but they are separate catalog entries. +- The canonical text form is `registry/repository:tag`, followed by `@digest` when the digest is known. Parsing the + canonical form gives back the same parts. + +### The catalog + +- A **registry repository** is a unique pair of registry (at most 200 characters) and repository (at most 300). +- An **image version** is a unique combination of registry repository, tag (at most 200 characters) and digest (at most + 255). The digest is optional; an image version without one is a different entry from the same tag with a digest. +- Registry, repository and tag must be non-blank. The catalog trims the digest and drops any `name@` prefix in front of + it, but changes no case: parts are compared exactly, so writers pass references normalized as above. +- Writers use get-or-create: an existing entry is returned unchanged; a new one stores the given publication time and + metadata (at most 4000 characters). Refreshing the publication time or metadata of an existing entry is done by the + component that owns the refresh ([Base images](base-images.md) for observed images and their bases, the runtime + container scan for container images, see [Update detection](update-detection.md)). +- **Concurrency:** when two writers create the same entry at once, the one whose insert collides with the uniqueness + rule re-reads and returns the entry the other created. No duplicates arise; no writer sees an error for the race. +- Every image version records which kind of component registered it last: observed image, runtime container or + base-image resolution. The value is informational and is overwritten by the next registering component. +- Catalog entries are shared: observed images, container snapshots, base-image relationships, update findings and + vulnerability findings all point at the same entries. Catalog entries are never deleted. + +### Live image versions + +- The live image versions are: the current version of every observed image (enabled or not); the image versions of + the containers in the latest runtime scan of each Docker instance (for an instance without a runtime scan, the + latest snapshot of each of its containers); and the base images recorded for any of these versions. +- The live set decides which images get vulnerability refreshes and which findings the cleanup keeps + ([Vulnerabilities](vulnerabilities.md), [Scan engine](scan-engine.md)). + +### Observed images + +- An observed image is a tracked target: a name (at most 200 characters, not required to be unique), an optional + description, its current image version, an enabled flag, its source and its creation and update times. +- Sources: *manual* (registered on `/observed-images`) and *discovery* (added by the Docker Hub account discovery and + shown as "My images" on `/my-images`; see [Registries](registries.md)). Discovered images are the user's own images. +- **Manual registration** takes a name and an image reference (both required) and an optional description, then: + 1. parses the reference and gets or creates its image version exactly as written — tag and, if given, digest — with + no registry call; + 2. looks for an observed image whose stored name equals the entered name exactly (case-sensitive, untrimmed); + 3. creates a new observed image (name trimmed, description trimmed or none, source *manual*, enabled), or updates + the found one: description replaced, current version replaced, enabled again; name and source stay; + 4. starts a manual scan of that image ([Base images](base-images.md)) and opens its detail page. +- The current version of a manual image never moves on its own: scans refresh its metadata but keep its tag and + digest. A discovered image's current version is moved by each discovery run to the repository's most recently + pushed tag. +- **Enabled state:** only discovery disables an observed image (when its repository disappears from the account) and + it enables again every image it lists. Manual registration of an existing name enables it again. There is no way to + disable or delete an observed image from the UI. Disabled images are skipped by the scheduled observed-image scan + ([Scan engine](scan-engine.md)) and keep their history. +- **Own images and running containers:** a running container is linked to an own image when its image's registry and + repository equal (case-insensitive) those of a discovered image's current version, regardless of tag and digest; + when several match, the one first by name wins. Manually registered images are not linked this way. + +## Limits and errors + +- A blank name or reference is rejected by the registration form. A reference without a repository name, or a value + longer than the stored limits, makes the registration fail; the form shows the error message and the observed image + is not saved (an image version created before the failure remains in the catalog). +- A name entered with leading or trailing whitespace does not match the stored, trimmed name and creates a further + observed image. +- The catalog rejects a blank registry, repository or tag. +- Nothing in this area handles credentials. + +## Related decisions + +None yet. + +## Not here + +- Reading tags and metadata from registries and the account discovery itself: [Registries](registries.md). +- Base-image chains and derived runtimes: [Base images](base-images.md). +- Comparing tags and choosing updates: [Update detection](update-detection.md). +- Container snapshots: [Docker instances](docker-instances.md). +- Vulnerability findings: [Vulnerabilities](vulnerabilities.md). +- Scan runs, schedules and retention: [Scan engine](scan-engine.md). +- What the pages show: [Web UI](web-ui.md). + +## Open questions + +1. Docker Hub references are not normalized to one form (`docker.io/nginx` vs `nginx`, `index.docker.io` vs + `docker.io`), so one image can have several catalog entries and not be recognized as shared. Should the parser + unify them? +2. Manual registration matches an existing observed image by the untrimmed, case-sensitive name and names are not + unique, so near-identical names create separate entries, and registering a manual image under a discovered image's + name updates the discovered one (which the next discovery run overwrites). Is name the intended identity? +3. Observed images can neither be disabled nor deleted by the user, and discovery re-enables every listed repository on + each run. Is a user-controlled enabled state planned? +4. The recorded "registered by" kind of an image version is last-writer-wins and a further kind (registry metadata) is + defined but never written. Is the field meant to carry meaning? diff --git a/docs/areas/registries.md b/docs/areas/registries.md new file mode 100644 index 0000000..51dd9af --- /dev/null +++ b/docs/areas/registries.md @@ -0,0 +1,204 @@ +# Registries + +## Scope + +Read access to container registries: Docker Hub (its web API and its registry) and every other registry that speaks the +OCI Distribution API. Covers which client serves which registry, what can be read (tags, manifests, image +configuration and labels), how DockerUpdateGuard authenticates, the Docker Hub account image discovery, the Docker Hub +request budget and parallelism, the retry of transient HTTP failures shared by the outbound HTTP integrations, and the +failure model every integration uses. The image catalog, base-image resolution and update detection rely on it for +registry data; operators rely on it to configure credentials and limits; the other integration areas link here for the +retry policy and the failure model. + +## Behavior + +### Failure model of integrations + +- An integration call answers with an outcome instead of raising an error for an expected failure. The outcomes are: + - *succeeded* — the call returned data; + - *not configured* — the feature lacks the configuration it needs; + - *unsupported* — the integration cannot serve this target (for example a registry no client handles); + - *not found* — the target does not exist (HTTP 404 for an account, repository, tag, manifest or blob); + - *failed* — the call was made and did not succeed (any other non-success status, a missing token, a response + without the expected content); + - *unknown* — the outcome could not be classified. +- Every outcome other than *succeeded* carries a human-readable message. Callers decide on the outcome, not on errors; + how a scan turns outcomes into scan-run statuses is described with each scan + ([Base images](base-images.md), [Update detection](update-detection.md), [Scan engine](scan-engine.md)). +- A *failed* outcome from a registry names the HTTP status code and includes the response body the remote side sent. + +### Choice of client per registry + +- The registry of an image reference (see [Image catalog](image-catalog.md)) decides which client serves it. The Docker + Hub client serves the hosts `docker.io`, `index.docker.io`, `registry-1.docker.io`, `hub.docker.com`, `docker.com` + and `www.docker.com`, compared case-insensitively; a registry given as an absolute URL is matched by its host. +- Every other non-empty registry is served by the generic OCI client. Only an empty registry is *unsupported*. +- The Docker Hub client uses three endpoints: the web API at `DockerUpdateGuard:DockerHub:ApiBaseUrl` (default + `https://hub.docker.com/`), the token service `https://auth.docker.io/token` and the registry + `https://registry-1.docker.io`. A repository without a namespace is addressed as `library/`. +- The OCI client addresses `https:///v2//…`. A registry given as an absolute URL is used with its + own scheme and base path; registry names are compared lowercased and keep a non-default port. +- Outbound requests to registries and release feeds carry the user agent `DockerUpdateGuard/1.0`. + +### What can be read + +- **Tag lookup** — digest and publication time of one tag. + - Docker Hub: the web API's tag record (the digest Docker Hub reports and its `last_pushed` time). + - Other registries: a manifest request for the tag. The digest is the `Docker-Content-Digest` of the first response + (for a multi-platform index, the index digest). The publication time is known only when the manifest carries a + top-level `created` timestamp; otherwise it is unknown. +- **Image configuration** — environment variables, labels (keys compared case-insensitively), creation time, operating + system and architecture, read from the image's configuration blob. For a multi-platform index one platform manifest + is chosen: + - Docker Hub: the first `linux` entry, skipping entries whose operating system or architecture is `unknown` + (attestations); otherwise the first remaining entry. + - Other registries: the entry matching the caller's operating system and architecture, then the first entry matching + the architecture, then the first entry with a digest. A caller that passes no platform gets the first entry. +- **Tag listing** — the tags of a repository with digest and publication time. + - Docker Hub: web API pages of 100 tags, in the order Docker Hub returns them (most recently pushed first), following + the `next` link of each page. + - Other registries: `tags/list` pages of 100 names following the `Link: rel="next"` header; every tag is then + inspected with a tag lookup. A tag whose lookup does not succeed is still listed, by name only. + +### Bounded tag listings + +- A caller may bound a listing with the current tag, the current digest, a minimum version tag, a version-line tag, a + published-since time and a maximum count (default 250 when bounds are given). Without bounds every tag is read. +- The current tag is always kept. With a version-line tag only tags of that version line are kept (version lines and + version comparison: [Update detection](update-detection.md)). A tag published before the published-since time, or + lower than the minimum version, is dropped unless its digest equals the current digest; a tag that cannot be + compared with the minimum version is kept. +- Docker Hub: the listing stops once the maximum count of kept tags is reached, and at the first tag older than the + published-since time — except when the current tag is `latest` with a known digest and no specific version tag + with that digest has been seen yet. +- Other registries: the maximum count limits the number of inspected tags; a tag that fails the name checks (current + tag, version line, minimum version) is not inspected. + +### Authentication + +- **Docker Hub web API.** Anonymous unless `DockerUpdateGuard:DockerHub:Pat` is set. With `Pat` and `UserName` (both + trimmed) the client logs in and sends the returned token as a bearer token. The token is kept in memory and renewed + one minute before its expiry (its `exp` claim, or five minutes after issue when it has none). `Pat` without + `UserName` makes every web API request *failed*, naming the missing key. `UserName` without `Pat` is anonymous. +- **Docker Hub registry.** A pull token is requested from the token service for every lookup, with the configured user + name and PAT as basic credentials when both are set, otherwise anonymously. +- **Other registries.** Access is anonymous only. On a 401 with a `Bearer` challenge a token is requested anonymously + from the challenge's realm, with its `service` and `scope` (default `repository::pull`). Tokens are + cached per registry host and repository until `issued_at` + `expires_in` (60 seconds when absent) minus 5 seconds; a + 401 with a cached token drops it and authenticates again once. Any other challenge, or a failed token request, makes + the call *failed* — private repositories outside Docker Hub cannot be read. + +### Docker Hub account discovery + +- Discovers the repositories of the configured Docker Hub account as own images. It runs only when both `UserName` + and `Pat` are set; otherwise each run is skipped with a log entry. It runs every + `DockerUpdateGuard:Scanning:DockerHubAccountDiscoveryIntervalMinutes` ([Scan engine](scan-engine.md)). +- The account listing requires `DockerUpdateGuard:DockerHub:Registry` to name a Docker Hub host; otherwise it answers + *unsupported* and the run ends without changes. +- The account is the namespace equal to `UserName`. All its repositories are listed (pages of 100), including private + ones the token can see. Repository names are lowercased; duplicates collapse to the most recently updated one. +- For each repository the full tag list is read and one tag is tracked: the most recently pushed; on equal push time + `latest` first, then alphabetical; tags without a push time come last. +- Each repository becomes an observed image with source *discovery*, matched by registry and repository + (case-insensitive). A new one is created enabled; an existing one gets its name set to the repository, its + description set to the Docker Hub description, its current version set to the tracked tag, and is enabled again. + The catalog entry is created as described in [Image catalog](image-catalog.md), with the tag's digest and push time. +- Discovered observed images whose repository is no longer listed are disabled, never deleted. +- A repository whose tag listing does not succeed or has no tags is skipped; its observed image keeps its state. When + the account listing does not succeed, nothing changes. +- All changes of a run are saved together at its end. Discovery does not scan; discovered images are scanned by the + observed-image scan ([Scan engine](scan-engine.md)). + +### Docker Hub request budget and parallelism + +- `Scanning:DockerHubRequestLimitWindowHours` (default 6, range 1–168), `Scanning:DockerHubRequestLimitPerWindow` + (default 200, range 1–100000) and `Scanning:DockerHubReservedManualRequestsPerWindow` (default 40, range 0–100000, + smaller than the per-window limit) describe the Docker Hub request budget. They only stretch the schedule of the + observed-image scan ([Scan engine](scan-engine.md)). Requests are not counted, manual scans are not limited, and no + request is refused for budget reasons. +- `DockerUpdateGuard:DockerHub:MaxParallelRequests` (default 4, range 1–32) caps the requests one Docker Hub client + instance has in flight at once, covering web API, login, token and registry requests. Every scan and every UI + operation works with its own client instances, so the cap is not global. The value is read when an instance is + created, so a configuration reload applies to later operations. Requests to other registries are not capped. +- Docker Hub rate limiting (HTTP 429) is handled by the retry below. + +### Retry of transient HTTP failures + +- Applies to Docker Hub (web API, login, token service, registry), other registries and their token services, the + .NET and nginx release feeds ([Base images](base-images.md)) and Portainer + ([Container actions](container-actions.md)). It does not apply to Docker Engine connections + ([Docker instances](docker-instances.md)) or to Docker Scout ([Vulnerabilities](vulnerabilities.md)). +- Transient are the responses 408, 429 and every 5xx, and transport errors and timeouts raised while sending. Other + statuses (including 401, 403 and 404) and cancellation are not retried. +- A request is retried up to `DockerUpdateGuard:Scanning:RetryCount` times (default 2, range 0–10; 0 disables + retries). The value is read for every request, so a configuration reload applies at once. +- The wait before a retry is the response's `Retry-After` (delta or date) when positive, capped at 30 seconds; + otherwise 500 ms × 2^n for the n-th retry (n starting at 0), capped at 30 seconds, plus a random jitter of up to 20 % + of that value, the total again capped at 30 seconds. +- Every method is retried, including requests with a body (the body is buffered so it can be resent). +- When the retries are used up the last response is returned to the caller, or the last error is raised. +- Each retry is logged as a warning with the method, the URL without its query, the status or error, the attempt + number and the wait. +- `DockerUpdateGuard:DockerHub:RequestTimeoutSeconds` (default 30, range 1–300) is the total time one request may + take including all retries and waits, for Docker Hub, other registries and the release feeds. A request that runs + out of time is not retried. + +### Configuration reload + +- Read live: `DockerHub:UserName`, `DockerHub:Pat`, `DockerHub:Registry` (for account discovery), + `DockerHub:MaxParallelRequests` (for new client instances) and `Scanning:RetryCount`. +- Fixed at startup, a change needs a restart: `DockerHub:ApiBaseUrl` and `DockerHub:RequestTimeoutSeconds`. +- Validation of these keys at startup (an invalid value fails startup, see [Hosting](hosting.md)): `Registry` + non-empty, `ApiBaseUrl` an absolute `http`/`https` URL, and the ranges above. + +## Limits and errors + +- *Unsupported*: an empty registry; the account listing when `DockerHub:Registry` is not a Docker Hub host. +- *Not found*: HTTP 404 for an account, repository, tag, manifest or configuration blob; an image whose manifest names + no configuration blob. +- *Failed*: any other non-success status, a token that cannot be obtained, a PAT without user name, a multi-platform + index without a usable entry. The message contains the status code and the remote response body. +- Some failures still surface as errors instead of outcomes: transport errors and timeouts that remain after the + retries, and malformed JSON, during tag lookups, tag listings and the account listing (both clients) and during + image configuration lookups on other registries. A Docker Hub configuration lookup turns them into *failed*. The + scan that made the call records such an error as a failed scan. +- The PAT is never written to logs or messages. The user name appears in the log entry and the message of a failed + Docker Hub login. +- Registry responses are untrusted: missing or mistyped JSON fields are treated as unknown values. An unbounded + listing (account discovery) reads every page the registry offers. + +## Related decisions + +None yet. + +## Not here + +- When scans and discovery run, and the formula that turns the request budget into a schedule: + [Scan engine](scan-engine.md). +- Parsing image references and storing what was read: [Image catalog](image-catalog.md). +- Following base-image labels and the release feeds' content: [Base images](base-images.md). +- Version lines, tag ranking and update recommendations: [Update detection](update-detection.md). +- Docker Scout and Trivy: [Vulnerabilities](vulnerabilities.md). +- Docker Engine access: [Docker instances](docker-instances.md); Portainer: [Container actions](container-actions.md). + +## Open questions + +1. `MaxParallelRequests` limits each Docker Hub client instance, not the process: concurrent scans, and the two client + instances one scan resolves, each get the full allowance. Is a process-wide cap intended? +2. The request budget assumes one Docker Hub request per observed image and scan, while a scan issues several (token, + manifest, platform manifest and blob per chain level, tag lookups and up to 150 tags per floating base tag). Should + the budget count real requests, or be documented as a rough scheduling hint only? +3. Requests with a body are retried like any other, including the Docker Hub login (and Portainer actions, see + [Container actions](container-actions.md)). Should non-idempotent requests be excluded? +4. `DockerHub:Registry` does not decide which registries the Docker Hub client serves (that host list is fixed). It + only gates the account listing and, when set to an absolute URL with a non-Docker-Hub host, replaces the web API + base address. What is the intended meaning of the key? +5. Transport errors, timeouts and malformed JSON escape as errors from several calls (see *Limits and errors*), which + contradicts the rule that integrations answer with outcomes. Should they map to *failed*? +6. For other registries an image configuration lookup without a platform takes the first index entry, which can be an + attestation manifest; Docker Hub skips `unknown` platforms. Should both behave alike? +7. Publication times for other registries come only from a top-level `created` field of the manifest, which most + registries do not send, so those tags usually have no publication time. Should the configuration blob's `created` + be used instead? +8. Pagination links (`next` on Docker Hub, `Link` elsewhere) are followed to whatever host they name, and the Docker + Hub bearer token is sent along. Should pagination be restricted to the original host? From a1d65a8405813f274ecaad781b32f56ceab7753e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 17:47:44 +0000 Subject: [PATCH 08/12] Link the area documents from ARCHITECTURE.md and correct stale claims Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/ARCHITECTURE.md | 466 +++++++++++++++++++------------------------ 1 file changed, 200 insertions(+), 266 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 390815e..1ef70bb 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,11 +2,13 @@ This document describes the architecture of DockerUpdateGuard as currently -implemented. It is a technical reference for contributors and AI coding -agents working in this repository; it complements, and does not repeat, -[README.md](../README.md) (product scope, configuration reference), the -[docs/](.) folder (contribution, testing, deployment tiers), and -[AGENTS.md](../AGENTS.md) / +implemented: how the parts fit together and why. It is a technical reference +for contributors and AI coding agents working in this repository. What each +part guarantees — formats, defaults, limits, error behavior — is written once +in the [area documents](areas/README.md) and linked from here instead of +repeated. It also complements [README.md](../README.md) (product scope, +configuration reference), the [docs/](.) folder (contribution, testing, +deployment tiers), and [AGENTS.md](../AGENTS.md) / [.github/copilot-instructions.md](../.github/copilot-instructions.md) and [.github/instructions/csharp.instructions.md](../.github/instructions/csharp.instructions.md) (coding conventions). @@ -15,19 +17,25 @@ agents working in this repository; it complements, and does not repeat, DockerUpdateGuard is a single-process ASP.NET Core web application that: -1. discovers Docker Engine instances and the containers running on them, +1. discovers Docker Engine instances and the containers running on them + ([Docker instances](areas/docker-instances.md)), 2. correlates what is running with registry tag/digest data to detect - available or needed updates, -3. resolves base-image relationships and reuse across observed images, -4. enriches images with vulnerability data (Trivy or Docker Scout), -5. records the results of all of the above as history in PostgreSQL, and -6. serves a Blazor Server UI over that data. + available or needed updates ([Registries](areas/registries.md), + [Image catalog](areas/image-catalog.md), + [Update detection](areas/update-detection.md)), +3. resolves base-image relationships and reuse across observed images + ([Base images](areas/base-images.md)), +4. enriches images with vulnerability data from Trivy or Docker Scout + ([Vulnerabilities](areas/vulnerabilities.md)), +5. records the results of all of the above as history in PostgreSQL + ([Scan engine](areas/scan-engine.md)), and +6. serves a Blazor Server UI over that data ([Web UI](areas/web-ui.md)). There is no separate API tier, message queue, or worker process — scanning, persistence, and UI all run inside one ASP.NET Core host, coordinated in-process via `IHostedService` background jobs and a shared PostgreSQL database. This keeps deployment to a single container plus PostgreSQL (see -§10), at the cost of the background scan engine and the UI sharing the same +§13), at the cost of the background scan engine and the UI sharing the same process and connection pool. ## 2. Solution layout @@ -75,84 +83,70 @@ documented in [AGENTS.md](../AGENTS.md) and not repeated here. registration to `ServiceCollectionExtensions.AddDockerUpdateGuardHost`, the composition root, which in order: -1. Binds and validates `DockerUpdateGuardOptions` (`.ValidateOnStart()` — - the host fails fast on invalid configuration before it accepts traffic). +1. Binds and validates `DockerUpdateGuardOptions` (`.ValidateOnStart()`). 2. Resolves the PostgreSQL connection string (`DockerUpdateGuardConnectionStringResolver`) and registers the EF Core - context via `DockerUpdateGuard.Data`'s `AddDockerUpdateGuardData`, with - `EnableRetryOnFailure` driven by `DatabaseOptions`. + context via `DockerUpdateGuard.Data`'s `AddDockerUpdateGuardData`. 3. Registers OpenTelemetry via `DockerUpdateGuard.Telemetry`'s `AddDockerUpdateGuardTelemetry`. -4. Registers `TransientHttpRetryHandler` as the outbound message handler - for every named `HttpClient` (Docker Hub, generic OCI registry, - .NET/nginx release metadata, Portainer). +4. Registers the typed/named `HttpClient`s (Docker Hub, generic OCI + registry, .NET/nginx release metadata, Portainer) with + `TransientHttpRetryHandler` as their outbound message handler. 5. Registers the integration clients, domain/orchestration services, the UI facade (`IApplicationViewService`), and all 7 background jobs - (`AddHostedService()`) — see §6 and §7. + (`AddHostedService()`) — see §6 to §8. `Program.cs` then adds MudBlazor (`AddMudServices()`) and Razor Components -with **Interactive Server** rendering (`AddInteractiveServerComponents()` / -`AddInteractiveServerRenderMode()`) — Blazor Server only, no WebAssembly or +with **Interactive Server** rendering — Blazor Server only, no WebAssembly or Auto render mode anywhere in the app. After `WebApplication.Build()` and before `RunAsync()`, the host calls `ApplicationInitializationExtensions.InitializeDockerUpdateGuardAsync`, -which blocks startup on: - -1. `DatabaseMigrator.MigrateAsync` — waits for the database to become - reachable (retrying transient connection failures), then, on PostgreSQL, - takes a session-scoped **advisory lock** - (`pg_advisory_lock(5410320540103205)`) before running EF Core migrations - and releases it in a `finally`. This serializes `dotnet ef`-style - migrations across multiple instances of the app starting concurrently - against the same database (e.g. a rolling deployment), so only one - instance applies migrations while the others wait. +which blocks startup on three steps: + +1. `DatabaseMigrator.MigrateAsync` — waits for the database, then applies + EF Core migrations under a PostgreSQL session-scoped advisory lock, so + concurrently starting instances never migrate at the same time. 2. `IInstanceDiscoveryService.SynchronizeConfiguredInstancesAsync` — - reconciles configured `DockerInstanceOptions` against the DB. This is - local, DB-only work and is awaited synchronously. + reconciles the configured Docker instances against the database + (local, DB-only work). 3. `ApplicationTelemetry.RefreshInventoryMetricsAsync` — seeds the - observable gauges (§9) with current counts. - -Docker Hub account discovery (`DockerHubAccountImageDiscoveryService`) is -**deliberately not** part of this blocking sequence, because it makes -sequential per-repository network calls — it instead runs as the first -iteration of `DockerHubAccountImageDiscoveryBackgroundService` immediately -after the host starts accepting traffic, so a slow or unreachable Docker -Hub does not delay application startup. - -The middleware pipeline (non-Development only where noted) is: -`UseExceptionHandler("/error")` → `UseHsts()` → `UseHttpsRedirection()` -(Production only) → `UseStatusCodePagesWithReExecute("/not-found")` → -`UseAntiforgery()` → `MapStaticAssets()` → -`MapRazorComponents().AddInteractiveServerRenderMode()`. There is no -authentication, authorization, health-check, or rate-limiting middleware — -see §11 for the resulting security posture. + inventory gauges (§10). + +Network-bound work (Docker Hub account discovery, scans) is deliberately +not part of this blocking sequence: it runs as the first iteration of the +background jobs once the host accepts traffic (§8), so a slow or +unreachable remote system does not delay startup. + +The exact startup order, retry and timeout values, the middleware pipeline +and error pages, and how configuration reload behaves are specified in +[Hosting and configuration](areas/hosting.md). There is no authentication, +authorization, health-check, or rate-limiting middleware (§11). ## 4. Configuration model Configuration is bound from the `DockerUpdateGuard` and `Telemetry` sections -via standard ASP.NET Core configuration (files, command-line, secret -stores). The full key-by-key reference (defaults, valid ranges, required -combinations) lives in [README.md](../README.md#configuration-reference) and -is not duplicated here. +via standard ASP.NET Core configuration (files, environment variables, +command line). The key-by-key reference lives in +[README.md](../README.md#configuration-reference); which keys are validated, +which apply on reload and which need a restart is specified in +[Hosting and configuration](areas/hosting.md), and the meaning of each +feature's keys in that feature's area. Architecturally relevant points: -- Every options class (`DockerUpdateGuardOptions` and its nested - `Database`/`DockerHub`/`ReleaseMetadata`/`Vulnerabilities`/`Scanning`/ - `DockerInstances[]`/`Portainer` sections, plus `TelemetryOptions`) has a - matching `IValidateOptions` validator wired with `.ValidateOnStart()`, - so misconfiguration is a startup failure, not a runtime surprise. -- `ScanningOptions` and `DockerHubOptions` are read live through - `IOptionsMonitor` by the background jobs and - `TransientHttpRetryHandler`, so interval/retry/parallelism changes in - `appsettings.json` take effect without a restart (a config *reload*, not - a live-edit UI). -- `DockerUpdateGuardConnectionStringResolver` prefers an inline - `DockerUpdateGuard:ConnectionString` over the named `ConnectionStrings` - entry, and is used identically by both the DI registration and the - validator so the two can never disagree about which connection string is - active. +- One `IValidateOptions` validator covers the whole `DockerUpdateGuard` + section and one the `Telemetry` section, both wired with + `.ValidateOnStart()`, so misconfiguration is a startup failure, not a + runtime surprise. +- The background jobs and `TransientHttpRetryHandler` read + `IOptionsMonitor` on every use, so schedule and + retry settings follow a configuration reload. Settings consumed while the + container is composed (`HttpClient` base addresses and timeouts, the + database, telemetry) are fixed until restart. +- `DockerUpdateGuardConnectionStringResolver` is used identically by the DI + registration and the validator, so the two can never disagree about which + connection string is active. ## 5. Data layer (`DockerUpdateGuard.Data`) @@ -163,25 +157,25 @@ auto-discovered via `ApplyConfigurationsFromAssembly`. ### 5.1 Entity model -| Entity | Role | -| --- | --- | -| `RegistryRepository` | Normalized `(Registry, Repository)` pair; parent of all `ImageVersion` rows for that repository. | -| `ImageVersion` | Normalized, deduplicated `(repository, tag, digest)` node — the central catalog entity; carries a vulnerability-assessment rollup. | -| `ImageRelationship` | Edge from a child `ImageVersion` to its base `ImageVersion` (`BaseImage`/`BuildStage`), with `Depth` — the base-image dependency graph. | -| `ObservedImage` | A user/discovery-registered "track this image" target; points at a `CurrentImageVersion`. | -| `DockerInstance` | A configured Docker Engine endpoint; optionally has one `PortainerEndpoint`. | -| `PortainerEndpoint` | Portainer connection details for a `DockerInstance`. | -| `ContainerSnapshot` | A discovered/running container at a point in time, with its resolved `UpdateAssessmentStatus`. | -| `ScanRun` | A persisted scan execution (type/status/trigger), optionally scoped to an `ObservedImage` or `DockerInstance`. | -| `UpdateFinding` | A detected update (subject `ImageVersion` → optional recommended `ImageVersion`), linked to an `ObservedImage` and/or `ContainerSnapshot`. | -| `TagCandidate` | One ranked candidate tag belonging to an `UpdateFinding`. | -| `VulnerabilityFinding` | A CVE/advisory against an `ImageVersion` (severity, fix status, CVSS). | -| `ContainerActionRun` | A requested Portainer action (restart/redeploy/update) against a container, service, or stack. | -| `RuntimeContainerTagSelection` | A manually pinned tag/digest override for a specific runtime container. | -| `DockerInstanceResourceSample` / `RuntimeContainerResourceSample` | Instance-level and per-container CPU/memory/network time series. | +| Entity | Role | Area | +| --- | --- | --- | +| `RegistryRepository` | `(Registry, Repository)` pair; parent of all `ImageVersion` rows for that repository. | [Image catalog](areas/image-catalog.md) | +| `ImageVersion` | Deduplicated `(repository, tag, digest)` node — the central catalog entity; carries a vulnerability-assessment rollup. | [Image catalog](areas/image-catalog.md) | +| `ImageRelationship` | Edge from a child `ImageVersion` to its base `ImageVersion`, with type and `Depth` — the base-image dependency graph. | [Base images](areas/base-images.md) | +| `ObservedImage` | A manually or discovery-registered "track this image" target; points at a `CurrentImageVersion`. | [Image catalog](areas/image-catalog.md) | +| `DockerInstance` | A configured Docker Engine endpoint; optionally has one `PortainerEndpoint`. | [Docker instances](areas/docker-instances.md) | +| `PortainerEndpoint` | Portainer connection details for a `DockerInstance`. | [Container actions](areas/container-actions.md) | +| `ContainerSnapshot` | A discovered container at a point in time, with its `UpdateAssessmentStatus`. | [Docker instances](areas/docker-instances.md) | +| `ScanRun` | A persisted scan execution (type/status/trigger), optionally scoped to an `ObservedImage` or `DockerInstance`. | [Scan engine](areas/scan-engine.md) | +| `UpdateFinding` | A detected update (subject `ImageVersion` → optional recommended `ImageVersion`), linked to an `ObservedImage` and/or `ContainerSnapshot`. | [Update detection](areas/update-detection.md) | +| `TagCandidate` | One ranked candidate tag belonging to an `UpdateFinding`. | [Update detection](areas/update-detection.md) | +| `VulnerabilityFinding` | A CVE/advisory against an `ImageVersion` (severity, fix status, CVSS). | [Vulnerabilities](areas/vulnerabilities.md) | +| `ContainerActionRun` | A record of a Portainer action against a container, service, or stack. | [Container actions](areas/container-actions.md) | +| `RuntimeContainerTagSelection` | A user's tag/digest selection for a specific runtime container. | [Update detection](areas/update-detection.md) | +| `DockerInstanceResourceSample` / `RuntimeContainerResourceSample` | Instance-level and per-container CPU/memory/network time series. | [Docker instances](areas/docker-instances.md) | Every enum used by these entities lives alongside them in the `Entities` -namespace (the dedicated `Enums\` folder is present but unused). +namespace. ### 5.2 Repositories vs. Queries @@ -189,20 +183,19 @@ The `Repositories\` and `Queries\` folders are deliberately split by read/write intent, not by entity: - **`Repositories\` — `IImageCatalogRepository`.** The single write-side - aggregate root for the `RegistryRepository`/`ImageVersion` catalog. Its - `GetOrCreateImageVersionAsync`/`GetOrCreateRegistryRepositoryAsync` - methods enforce catalog uniqueness (including a `DbUpdateException` - race-retry path for concurrent get-or-create calls under load) so - scanners never create duplicate catalog rows. This is the only - repository in the project — no entity outside the catalog needed one. + entry point for the `RegistryRepository`/`ImageVersion` catalog. Its + get-or-create methods enforce catalog uniqueness, including a race-retry + path for concurrent get-or-create calls, so scanners never create + duplicate catalog rows ([Image catalog](areas/image-catalog.md)). This is + the only repository in the project — no entity outside the catalog + needed one. - **`Queries\` — `ILiveImageInventoryQueryService`, `ISharedBaseImageQueryService`.** Read-only, multi-entity, page-shaped - projections (joins/grouping across `ImageVersion`, `ImageRelationship`, - `ObservedImage`, `ContainerSnapshot`) that return plain DTOs instead of - tracked entities, and don't naturally belong to any single entity's - repository. `ILiveImageInventoryQueryService.GetLiveImageVersionIdsAsync` - in particular is the shared definition of "still relevant" used by - `ScanCleanupBackgroundService` (§7) to decide what history to keep. + projections that return plain DTOs instead of tracked entities and don't + naturally belong to any single entity's repository. + `ILiveImageInventoryQueryService` is the shared definition of "still + relevant" that cleanup uses to decide what history to keep + ([Scan engine](areas/scan-engine.md)). ### 5.3 Migrations @@ -216,34 +209,34 @@ DockerUpdateGuardDbContextFactory` provides the design-time `DbContext` for Each external system has one hand-written client behind a narrow interface — there is no dependency on `Docker.DotNet`, a Portainer SDK, or a Trivy -client library. All clients return plain DTOs wrapped in +client library. Clients return plain DTOs wrapped in `ExternalOperationResult` (`Infrastructure\ExternalOperationResult{T}.cs`) -rather than throwing for expected failure modes (`NotConfigured`, -`Unsupported`, `NotFound`, `Failed`, `Unknown`); orchestrators branch on -that status instead of on exceptions. +for expected failure modes, and orchestrators branch on that status instead +of on exceptions. The outcomes and where errors still escape are specified +in [Registries](areas/registries.md#failure-model-of-integrations). -| Client | Talks to | Notes | +| Client | Talks to | Area | | --- | --- | --- | -| `DockerInstanceClient` (`Docker\`) | Docker Engine HTTP API (`v1.41`) | One HTTP transport supporting `http(s)://`, `tcp://`, `unix://` (via `SocketsHttpHandler.ConnectCallback`), and `npipe://` (Windows named pipes) behind a single client, pooled per connection config. | -| `DockerHubClient` (`DockerHub\`) | Docker Hub REST API **and** the OCI Distribution v2 API (`registry-1.docker.io`) | Implements both `IDockerHubClient` (repository/tag metadata) and the generic `IRegistryMetadataClient` (manifest/blob/base-image-label resolution), because Docker Hub is both a metadata API and an OCI registry. | -| `OciRegistryClient` (`Images\`) | Any OCI Distribution v2 registry | The registry-agnostic counterpart to `DockerHubClient`'s registry half, for non-Docker-Hub registries. | -| `PortainerClient` (`Portainer\`) | Portainer REST API | JWT login, then endpoint/container listing and action execution (restart/redeploy/update) behind `IPortainerClient`. | -| `TrivyVulnerabilityProvider` (`Vulnerabilities\`) | Trivy CLI (external process), client mode against a Trivy server | Runs `trivy image --server --format json …` via `IProcessRunner`/`ProcessRunner`, parses stdout JSON. The `trivy` binary ships inside the container image (see §10). | -| `DockerScoutVulnerabilityProvider` (`Vulnerabilities\`) | Docker Scout / Docker Hub HTTP API | Docker Hub login → cached bearer token → paginated vulnerability fetch (capped at 50 pages). | -| `DefaultVulnerabilityProvider` (`Vulnerabilities\`) | — | No-op fallback when vulnerability scanning is disabled/unconfigured (`IVulnerabilityProviderResolver` selects it). | +| `DockerInstanceClient` (`Docker\`) | Docker Engine HTTP API over `http(s)://`, `tcp://`, `unix://` and `npipe://`, one hand-built `HttpClient` pooled per connection config. | [Docker instances](areas/docker-instances.md) | +| `DockerHubClient` (`DockerHub\`) | Docker Hub REST API **and** the OCI Distribution v2 API (`registry-1.docker.io`); implements both `IDockerHubClient` and the generic `IRegistryMetadataClient`, because Docker Hub is both a metadata API and an OCI registry. | [Registries](areas/registries.md) | +| `OciRegistryClient` (`Images\`) | Any other OCI Distribution v2 registry. | [Registries](areas/registries.md) | +| `PortainerClient` (`Portainer\`) | Portainer REST API (API key or JWT login), endpoint and container listing, container actions. | [Container actions](areas/container-actions.md) | +| `TrivyVulnerabilityProvider` (`Vulnerabilities\`) | Trivy CLI (external process via `IProcessRunner`), client mode against a Trivy server. | [Vulnerabilities](areas/vulnerabilities.md) | +| `DockerScoutVulnerabilityProvider` (`Vulnerabilities\`) | Docker Scout / Docker Hub HTTP API. | [Vulnerabilities](areas/vulnerabilities.md) | +| `DefaultVulnerabilityProvider` (`Vulnerabilities\`) | — (no-op fallback selected by `IVulnerabilityProviderResolver` when scanning is disabled). | [Vulnerabilities](areas/vulnerabilities.md) | `RegistryMetadataService` is the dispatcher in front of `IRegistryMetadataClient`: it holds every registered client and picks the first whose `CanHandle(registry)` matches, falling back to -`NullRegistryMetadataClient` (an `Unsupported` sentinel) for registries -nothing can handle — adding a new registry type means adding one more -`IRegistryMetadataClient` implementation, not touching the dispatcher. +`NullRegistryMetadataClient` (an `Unsupported` sentinel) — adding a new +registry type means adding one more `IRegistryMetadataClient` +implementation, not touching the dispatcher. -`TransientHttpRetryHandler` (`Infrastructure\`) is attached to every -outbound `HttpClient` (Docker Hub, OCI registry, release-metadata feeds, -Portainer) and retries transient failures (408/429/5xx, -`HttpRequestException`, `TimeoutException`) honoring `Retry-After`, -otherwise exponential backoff with jitter, up to `ScanningOptions.RetryCount`. +`TransientHttpRetryHandler` (`Infrastructure\`) is attached to the +typed/named `HttpClient`s registered in the composition root (§3). The +Docker Engine client and Docker Scout create their `HttpClient`s without +it. The retry policy is specified in +[Registries](areas/registries.md#retry-of-transient-http-failures). ## 7. Domain / orchestration layer and the scan engine @@ -266,7 +259,7 @@ flowchart TD ISO[ImageScanOrchestrator] RCSO[RuntimeContainerScanOrchestrator] VES[VulnerabilityEnrichmentService] - BIR["IBaseImageResolver
(DockerHub/Registry)"] + BIR["IBaseImageResolver
(registry labels)"] end subgraph Persistence["DockerUpdateGuard.Data"] @@ -287,35 +280,24 @@ flowchart TD ``` - **`IUpdateDetectionService` (`UpdateDetectionService`)** is a pure - evaluator (`Evaluate(ImageReference, tags) -> UpdateEvaluationResult`, - no I/O), so update logic can be unit-tested without mocking HTTP or the - database. -- **`VersionTagResolutionHelper`** is the tag-comparison engine: parses - `major.minor.patch[-suffix]` and `YYYY-suffix` (year-prefixed) tags, - groups tags into "variant families" (e.g. `-alpine`, `-jammy`) so an - Alpine tag is never recommended as an update for a Debian one, resolves - floating alias tags (`latest`) back to the concrete semver tag sharing - their digest, and ranks pre-release identifiers below GA releases. -- **Major-version upgrades are deliberately conservative.** A new major - line is only surfaced as an actionable update once it has at least - `Scanning:MajorUpgradeMinimumReleaseCount` releases, its first (`x.0.0`) - release is older than `Scanning:MajorUpgradeMinimumAgeDays`, and the - current major line has gone dormant. Until then it is shown only as - descriptive text on an otherwise `UpToDate` result — this avoids - recommending a major upgrade the day it ships. Both thresholds accept - `0` to disable the corresponding wait. -- **`IDerivedBaseRuntimeDetector`** infers a locally running .NET/nginx - runtime version from Docker inspect environment variables and image - history alone (no registry call), which `IDotNetReleaseMetadataService`/ - `INginxReleaseMetadataService` then compare against upstream release - feeds — this covers images that don't expose a useful base-image label, - at the cost of only recognizing the two runtimes it has explicit - detection rules for. + evaluator (no network or database I/O) over an image reference and its + tags, and **`VersionTagResolutionHelper`** is the tag-comparison engine + behind it, so update logic — including the deliberately conservative + major-upgrade gating — can be unit-tested without mocking HTTP or the + database. The rules themselves are in + [Update detection](areas/update-detection.md). +- **`IBaseImageResolver`** (registered as `RegistryBaseImageResolver`) + resolves base images from registry image labels; **`IDerivedBaseRuntimeDetector`** + infers a .NET/nginx runtime from environment variables (and, for runtime + containers, image history), which `IDotNetReleaseMetadataService`/ + `INginxReleaseMetadataService` compare against upstream release feeds. + This covers images without a useful base-image label, at the cost of only + recognizing the two runtimes it has explicit rules for + ([Base images](areas/base-images.md)). - **`ImageScanOrchestrator`** guards against overlapping scans of the same - observed image with a static per-image `SemaphoreSlim` registry - (`ObservedImageScanLockEntry`), not a database lock — this is a - same-process concurrency guard, not a multi-instance one (see §11 for - the deployment assumption this relies on). + observed image with a static per-image `SemaphoreSlim` registry, not a + database lock — a same-process concurrency guard, not a multi-instance + one (§11, [Scan engine](areas/scan-engine.md)). - Orchestrators write directly to `DbContext` sets (`UpdateFindings`, `ImageRelationships`, `ScanRuns`) for their own aggregates, and go through `IImageCatalogRepository` only for the shared, dedup-sensitive @@ -329,52 +311,26 @@ All seven scheduled jobs derive from `ScheduledBackgroundService` `ExecuteCoreAsync`, optionally `ExecuteStartupAsync` (runs once before the loop) and override `ShouldExecuteImmediately()` (default `true`). The base class re-evaluates `GetInterval()` every iteration (so config-reload -changes apply live), creates a DI scope per run, and catches/logs -exceptions from `ExecuteCoreAsync` so one failing run never crashes the -host. +changes apply live), creates a DI scope per run, and catches and logs +exceptions from `ExecuteCoreAsync`, so a failing run does not end the job. -| Job | Does | Interval source | -| --- | --- | --- | -| `DockerInstanceDiscoveryBackgroundService` | Reconciles configured `DockerInstanceOptions` against the DB. | `Scanning:DiscoveryIntervalMinutes` | -| `DockerHubAccountImageDiscoveryBackgroundService` | Discovers repositories/images under the configured Docker Hub account. Runs immediately at startup (§3). | `Scanning:DockerHubAccountDiscoveryIntervalMinutes` | -| `OwnImageBaseRefreshBackgroundService` | Scans all enabled observed images for base-image/tag updates (`ImageScanOrchestrator.ScanAllAsync`). | **Dynamic** — see below | -| `RuntimeContainerRefreshBackgroundService` | Scans running containers for update opportunities (`RuntimeContainerScanOrchestrator.ScanAllAsync`). | `Scanning:RuntimeImageUpdateScanIntervalMinutes` | -| `ResourceStatisticsRefreshBackgroundService` | Samples Docker instance/container CPU/memory/network usage. | `Scanning:ResourceStatisticsIntervalMinutes` | -| `VulnerabilityRefreshBackgroundService` | Refreshes CVE findings via the resolved `IVulnerabilityProvider`. | `Scanning:VulnerabilityRefreshIntervalMinutes` | -| `ScanCleanupBackgroundService` | Repairs stale `Running` scan rows on startup, then purges history past retention (see below). | `Scanning:CleanupIntervalMinutes` | - -**Dynamic own-image scan interval.** Unlike the other jobs, -`OwnImageBaseRefreshBackgroundService.GetInterval()` computes its interval -per iteration via `ObservedImageScanIntervalCalculator`: -`max(OwnImageBaseScanIntervalMinutes, ⌈(DockerHubRequestLimitWindowHours × -60 × enabledObservedImageCount) / (DockerHubRequestLimitPerWindow − -DockerHubReservedManualRequestsPerWindow)⌉)`. As the number of tracked -images grows, the interval stretches so scheduled Docker Hub calls stay -within the configured request budget for the window while always reserving -`DockerHubReservedManualRequestsPerWindow` capacity for manual/ad-hoc -scans. This is one of three independent Docker Hub rate-limiting -mechanisms: - -1. **Schedule-level** — the dynamic interval above. -2. **Concurrency-level** — `DockerHubClient` bounds concurrent outbound - calls with a `SemaphoreSlim` sized to `DockerHub:MaxParallelRequests`. -3. **Transport-level** — `TransientHttpRetryHandler` (§6) treats HTTP 429 - as transient and retries with backoff/`Retry-After`. - -There is no separate token-bucket or sliding-window counter; the three -mechanisms above compose to keep Docker Hub traffic within budget. - -**Cleanup and crash recovery.** `ScanCleanupBackgroundService` does not run -immediately; its `ExecuteStartupAsync` first repairs any `ScanRun` stuck in -`Running` status (from a crash or restart) older than -`max(2 × the relevant scan interval, 24h)`, marking it `Failed`. Its -recurring pass then purges data older than `Scanning:RetainScanRunsDays`: -resolved/stale `TagCandidates`, resolved `UpdateFindings`, dead -`VulnerabilityFindings` (immediately if the image is no longer "live" per -`ILiveImageInventoryQueryService`, else age-based), orphaned -`ContainerSnapshots`, and old resource samples — while always keeping the -most recent 20 completed `ScanRun`s for the Scan History UI even past -retention. It finishes by refreshing the telemetry gauges (§9). +| Job | Does | +| --- | --- | +| `DockerInstanceDiscoveryBackgroundService` | Reconciles configured Docker instances against the DB. | +| `DockerHubAccountImageDiscoveryBackgroundService` | Discovers repositories/images under the configured Docker Hub account. | +| `OwnImageBaseRefreshBackgroundService` | Scans all enabled observed images (`ImageScanOrchestrator.ScanAllAsync`) on a dynamic interval. | +| `RuntimeContainerRefreshBackgroundService` | Scans running containers (`RuntimeContainerScanOrchestrator.ScanAllAsync`). | +| `ResourceStatisticsRefreshBackgroundService` | Samples Docker instance/container CPU/memory/network usage. | +| `VulnerabilityRefreshBackgroundService` | Refreshes CVE findings via the resolved `IVulnerabilityProvider`. | +| `ScanCleanupBackgroundService` | Repairs stale `Running` scan rows on startup, then purges history past retention. | + +Intervals and their configuration keys, the dynamic own-image interval that +stretches with the number of observed images to stay within the Docker Hub +request budget, overlap rules, scan-run statuses, crash recovery and +retention are specified in [Scan engine](areas/scan-engine.md). Docker Hub +traffic is shaped by that schedule, by a per-client concurrency limit and +by the transport retry (§6); none of them counts requests. Their exact +behavior is in [Registries](areas/registries.md). ## 9. UI layer @@ -382,92 +338,68 @@ Blazor Server (**Interactive Server** render mode, applied at the `App` root — no WebAssembly, no Auto mode) with **MudBlazor** as the component library, plus a hand-rolled CSS design-system layer (`wwwroot\app.css`, light/dark via a `.dug-dark` class) on top of it. +Pages, routes and what each page shows or triggers are specified in +[Web UI](areas/web-ui.md). -- **Pages** (`Components\Pages`, one `@page` route each): Dashboard (`/`), - My Images / My Image Detail, Observed Images / Observed Image Detail, - Runtime Containers / Runtime Container Detail, Docker Instances / Docker - Instance Detail, Shared Base Images (`/base-images`), Vulnerabilities, - Scan History, plus `Error`/`NotFound`. +- **Pages** (`Components\Pages`) have one `@page` route each. - **`IApplicationViewService` (`UI\ApplicationViewService`)** is the single facade every page depends on. It maps `DockerUpdateGuard.Data` entities - into a set of `*ViewData` DTOs (`UI\`) — pages never query + into `*ViewData` DTOs (`UI\`) — pages never query `DockerUpdateGuardDbContext` or the repositories/query services directly. This keeps EF Core change-tracking and entity shapes out of the component tree, at the cost of `ApplicationViewService` being a fairly wide surface (one method per page/section). - **`Components\Shared`** holds cross-page building blocks: loading - skeletons (`PageHeaderSkeleton`, `TableSkeleton`, `MetricGridSkeleton`) - and the vulnerability widgets (`VulnerabilityAssessmentCard`, - `VulnerabilityFindingsTable`, `VulnerabilitySeverityChips`, - `VulnerabilityRescanControl`) shared between the Vulnerabilities page and - image/container detail pages. -- **`DashboardRefreshState`** is a small injected scoped pub/sub class - background jobs and manual-scan actions use to signal the Dashboard to - refresh without a full page reload. + skeletons and the vulnerability widgets shared between the + Vulnerabilities page and image/container detail pages. +- **`DashboardRefreshState`** is a small singleton pub/sub class that + manual-scan actions in the UI use to signal open Dashboards to refresh + without a full page reload. ## 10. Telemetry `DockerUpdateGuard.Telemetry.AddDockerUpdateGuardTelemetry` wires -OpenTelemetry logging, tracing, and metrics — each independently -toggleable (`Telemetry:EnableLogging/EnableMetrics/EnableTracing`, default -on) — exported over a single OTLP exporter when `Telemetry:OtlpEndpoint` is -a valid absolute HTTP(S) URI (registration is skipped entirely if all three -signals are disabled; there is no console/Prometheus exporter fallback). - -- **Tracing**: a custom `ActivitySource` (`DockerUpdateGuardTelemetry. - ActivitySource`) plus ASP.NET Core and HttpClient instrumentation. - `TelemetryActivityNames` defines the custom activity names in use - (`scan.run`, `dockerhub.request`, `dockerengine.request`, `cve.request`, - `portainer.request`, `portainer.action`, `persistence.operation`). -- **Metrics**: a custom `Meter` plus ASP.NET Core, HttpClient, and .NET - runtime instrumentation. `ApplicationTelemetry` (host project) owns the - application-specific instruments: counters/histogram for scan - runs/failures/duration, and observable gauges for inventory counts - (`ObservedImages`, `RuntimeContainers`, `DeduplicatedBaseImages`, - `ActiveUpdateFindings`, `ActiveCveFindings`, `NeedsReviewFindings`) — - refreshed at startup and after every cleanup pass (§8). -- **Logging**: structured, via `ILogger` and source-generated - `[LoggerMessage]` partial classes — every integration client and - background job has a matching `*Logging.cs` partial class following this - pattern, rather than ad-hoc `_logger.LogInformation(...)` calls. +OpenTelemetry logging, tracing, and metrics, exported over a single OTLP +exporter. A custom `ActivitySource` and `Meter` live in the telemetry +project (`DockerUpdateGuardTelemetry`, `TelemetryActivityNames`); +`ApplicationTelemetry` in the host project owns the application-specific +instruments. Logging is structured, via `ILogger` and source-generated +`[LoggerMessage]` partial classes — every integration client and +background job has a matching `*Logging.cs` partial class rather than +ad-hoc `_logger.LogInformation(...)` calls. Switches, export conditions, +the emitted span and instrument names, and when the inventory gauges +refresh are specified in [Telemetry](areas/telemetry.md). ## 11. Security posture and operating assumptions -These are observed characteristics of the current implementation, not -aspirational guarantees: +These are observed characteristics of the current implementation; the +details are in [Hosting and configuration](areas/hosting.md) and the areas +it links: -- **No authentication or authorization layer.** There is no - `AddAuthentication`/`AddAuthorization`, no login page, and no API key - check anywhere in the host. The application is meant to be deployed - behind network-level access control (private network, VPN, or a reverse - proxy that terminates auth) rather than exposed directly — this is a - deployment responsibility, not something the app enforces itself. See +- **No authentication or authorization layer.** The application is meant + to be deployed behind network-level access control (private network, + VPN, or a reverse proxy that terminates auth) — a deployment + responsibility, not something the app enforces. See [SECURITY.md](../SECURITY.md) for the vulnerability-reporting process. - **No health-check or rate-limiting endpoints** are registered. -- **Single-instance assumption for the scan engine.** The PostgreSQL - advisory lock (§3) serializes *migrations* across concurrently starting - instances, but the background jobs and `ImageScanOrchestrator`'s - in-memory scan lock (§7) are not distributed — running more than one - instance of the app against the same database will duplicate scan work - (harmless but wasteful; catalog writes stay correct via - `IImageCatalogRepository`'s get-or-create race-retry) rather than - coordinate it. -- **Container runs as a non-root, unprivileged user** (UID 64000, GID 0 — - the root group, not root user) so a group-owned, root:root Docker socket - bind-mount stays readable (e.g. on Synology DSM) without running the - process as root; see the comment block in +- **Single-instance assumption for the scan engine.** The advisory lock + (§3) serializes *migrations* across concurrently starting instances, but + the background jobs and `ImageScanOrchestrator`'s in-memory scan lock + (§7) are not distributed — running more than one instance against the + same database duplicates scan work rather than coordinating it; catalog + writes stay correct via `IImageCatalogRepository`'s race-retry. +- **Container runs as a non-root user** (UID 64000, GID 0) so a + group-owned, root:root Docker socket bind-mount stays readable without + running the process as root; see [src/DockerUpdateGuard/Dockerfile](../src/DockerUpdateGuard/Dockerfile). - Mounting the host's Docker socket is nonetheless a high-trust operation - (equivalent to root on the host) and is inherent to the feature, not a - bug. + Mounting the host's Docker socket is nonetheless root-equivalent on the + host and inherent to the feature. - **Secrets in configuration.** Docker Hub PAT, Portainer credentials/API - token, and the database password are plain configuration values (no - secret-store integration beyond what ASP.NET Core's configuration - providers already support). `SkipCertificateValidation` (Docker - instances) and `AllowInsecureHttp` (Portainer) exist for - trusted/self-signed/local setups and are logged as warnings at runtime - when active — operators are responsible for not enabling them over - untrusted networks. + token, and the database password are plain configuration values. + `SkipCertificateValidation` (Docker instances) and `AllowInsecureHttp` + (Portainer) exist for trusted/self-signed/local setups; how each is + validated and reported is in [Docker instances](areas/docker-instances.md) + and [Container actions](areas/container-actions.md). ## 12. Testing architecture @@ -497,14 +429,16 @@ the two production projects that matter for this split: `v*.*.*` tag — builds, tests, then builds and pushes the multi-stage Docker image (tagged with the version and `latest`) to Docker Hub (`networlddev/dockerupdateguard`) and creates a GitHub Release with - auto-generated notes. The image build records the exact - `mcr.microsoft.com/dotnet/runtime` base digest as an OCI label. + auto-generated notes. The workflow passes the digest of + `mcr.microsoft.com/dotnet/runtime:10.0-alpine` into the image's + `org.opencontainers.image.base.digest` label, while the runtime stage + builds on `mcr.microsoft.com/dotnet/aspnet:10.0-alpine`. - **Container image** (`src/DockerUpdateGuard/Dockerfile`): multi-stage — SDK build stage, - a `trivy` CLI stage copied in from the official Trivy image (bundling - the CLI so `Vulnerabilities:Provider=Trivy` works out of the box against - a separately-run Trivy *server*), and an Alpine ASP.NET runtime stage - running as the non-root user described in §11. Listens on `:8080`. + a `trivy` CLI stage copied in from the official Trivy image (so the + Trivy provider works out of the box against a separately-run Trivy + *server*), and an Alpine ASP.NET runtime stage running as the non-root + user described in §11; see [Hosting and configuration](areas/hosting.md). - **Deployment tiers** (`docs/docker-compose.{minimal,trivy,full}.yml`, walked through in [INSTALL.md](../INSTALL.md)): minimal (app + PostgreSQL), + Trivy server, or + a full Grafana OTel-LGTM stack @@ -520,9 +454,9 @@ Summarized here for quick reference; details are in the sections above. | Decision | Rationale evidenced in the code | | --- | --- | | Blazor Server (Interactive Server), not WebAssembly | Keeps all data access server-side against `DockerUpdateGuardDbContext`; no separate public API surface is needed since there is no WASM client to serve one to. | -| Hand-written HTTP clients instead of `Docker.DotNet`/a Portainer SDK/a Trivy client library | Each client only needs a small slice of its target API (engine stats/inspect, Hub tags, Portainer actions, Trivy CLI JSON) — see §6 for exactly what each one covers. | +| Hand-written HTTP clients instead of `Docker.DotNet`/a Portainer SDK/a Trivy client library | Each client only needs a small slice of its target API — see §6 for what each one covers. | | Trivy runs as an external CLI process (`IProcessRunner`), not an HTTP client to a Trivy server API | The image is scanned locally by the CLI in client mode; only the vulnerability database lookup goes to the configured Trivy server. | -| Update detection (`IUpdateDetectionService`) and tag comparison (`VersionTagResolutionHelper`) are pure, I/O-free | Lets the update-recommendation logic — including the deliberately conservative major-version-upgrade gating (§7) — be unit-tested exhaustively without mocking HTTP or the database. | +| Update detection (`IUpdateDetectionService`) and tag comparison (`VersionTagResolutionHelper`) are pure, I/O-free | Lets the update-recommendation logic — including the deliberately conservative major-upgrade gating — be unit-tested exhaustively without mocking HTTP or the database. | | One shared `IImageCatalogRepository` for the `RegistryRepository`/`ImageVersion` catalog, but no repository for any other entity | Only the catalog has a get-or-create-with-dedup concern that benefits from a single enforced entry point (§5.2); other entities are written directly by the orchestrator that owns them. | | PostgreSQL advisory lock around migrations, but no distributed lock around scanning | Migrations are unsafe to run concurrently and are cheap to serialize once at startup; duplicate concurrent scans are wasteful but not unsafe (§11), so they were not made distributed. | | No authentication/authorization in the app itself | The app assumes a trusted network perimeter (reverse proxy, VPN, private network) rather than implementing its own auth layer (§11). | From c51d68d438aa2e6ff8e2dc8191c5b5c823a744c4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 20:08:32 +0000 Subject: [PATCH 09/12] Link the open issues from the area documents Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/ARCHITECTURE.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 1ef70bb..e920885 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -432,7 +432,8 @@ the two production projects that matter for this split: auto-generated notes. The workflow passes the digest of `mcr.microsoft.com/dotnet/runtime:10.0-alpine` into the image's `org.opencontainers.image.base.digest` label, while the runtime stage - builds on `mcr.microsoft.com/dotnet/aspnet:10.0-alpine`. + builds on `mcr.microsoft.com/dotnet/aspnet:10.0-alpine` + ([#37](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/37)). - **Container image** (`src/DockerUpdateGuard/Dockerfile`): multi-stage — SDK build stage, a `trivy` CLI stage copied in from the official Trivy image (so the From 7ff72570e6f2513bf5c2443012f697f5d39f0e99 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 20:09:15 +0000 Subject: [PATCH 10/12] Resolve the open questions of the image areas Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/base-images.md | 26 ++++++++++++-------------- docs/areas/image-catalog.md | 28 +++++++++++++++------------- docs/areas/update-detection.md | 23 +++++++++++++---------- 3 files changed, 40 insertions(+), 37 deletions(-) diff --git a/docs/areas/base-images.md b/docs/areas/base-images.md index fc6d55f..640593e 100644 --- a/docs/areas/base-images.md +++ b/docs/areas/base-images.md @@ -25,8 +25,9 @@ a base-image update or vulnerability reaches; vulnerability refreshes and the da ### Resolving the chain -- Base images are resolved only for observed images. Running containers are not resolved themselves; they share an - observed image's chain only when they run exactly that image version. +- Base images are resolved only for observed images. Running containers are not resolved themselves; a container + joins an observed image's chain only when it runs the exact image version — tag and digest — of that observed image. + A container on another digest of the same tag has no base images. - The source is the image configuration's labels: `org.opencontainers.image.base.name` names the base image (parsed as an image reference, see [Image catalog](image-catalog.md)); `org.opencontainers.image.base.digest` supplies its digest (lowercased) when the name carries none. The same rule applies to Docker Hub and to every other registry. @@ -75,7 +76,8 @@ a base-image update or vulnerability reaches; vulnerability refreshes and the da - Recognizes the .NET or nginx runtime an image is built on, also without base-image labels. One runtime per image. - Inputs: for observed images the environment variables of the image configuration from the registry; for running containers the environment of the local image and its build history (each entry's command and comment) from the - Docker Engine ([Docker instances](docker-instances.md)). + Docker Engine ([Docker instances](docker-instances.md)). Registry images have no build history, so the + history-based signals apply to running containers only. - Signals, strongest first; on equal strength environment variables win over history, and earlier history entries over later ones: @@ -122,6 +124,13 @@ a base-image update or vulnerability reaches; vulnerability refreshes and the da - Feed failures (non-success status, missing `releases-index` array, unreadable content) only suppress the runtime finding; they do not change the scan status. Unreadable content is logged as a warning. +### Known gaps + +- The relationship type for build stages is defined in storage but never produced. Tracked in + [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). +- The shared base images view also lists bases used by a single observed image and counts disabled observed images. + Tracked in [#286](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/286). + ## Related decisions None yet. @@ -134,14 +143,3 @@ None yet. - Vulnerabilities of base images: [Vulnerabilities](vulnerabilities.md). - Schedules, overlap rules and scan-run retention: [Scan engine](scan-engine.md). - Page layout of `/base-images` and the detail pages: [Web UI](web-ui.md). - -## Open questions - -1. Running containers are only connected to a base-image chain when they run the very image version an observed image - points at (same tag and digest); a container on a different digest of the same tag shows no base images. Should - containers be resolved, or matched by repository? -2. The build-stage relationship type is defined but never produced. Should it be removed or implemented? -3. The shared base images view lists bases used by a single observed image and counts disabled observed images. Is - "shared" meant to require at least two users, and should disabled images count? -4. Derived runtime detection for observed images uses environment variables only, without build history, so the - history-based signals apply only to running containers. Intended? diff --git a/docs/areas/image-catalog.md b/docs/areas/image-catalog.md index 7599398..dc19e8d 100644 --- a/docs/areas/image-catalog.md +++ b/docs/areas/image-catalog.md @@ -91,6 +91,21 @@ refer to the same image the same way. - The catalog rejects a blank registry, repository or tag. - Nothing in this area handles credentials. +### Known gaps + +- Docker Hub references are not normalized to one form (`docker.io/nginx` vs `nginx`, `index.docker.io` vs + `docker.io`), so one image can have several catalog entries and is then not recognized as shared. Tracked in + [#275](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/275). +- Manual registration identifies an existing observed image by its untrimmed, case-sensitive name, and names are not + unique: near-identical names create separate entries, and registering a manual image under a discovered image's name + updates the discovered one, which the next discovery run overwrites. + Tracked in [#284](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/284). +- Observed images cannot be disabled or deleted by the user, and discovery enables every listed repository again on + each run. Tracked in [#282](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/282). +- The recorded kind of the component that last registered an image version is last-writer-wins and carries no + meaning, and a further kind (registry metadata) is defined but never written. + Tracked in [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). + ## Related decisions None yet. @@ -104,16 +119,3 @@ None yet. - Vulnerability findings: [Vulnerabilities](vulnerabilities.md). - Scan runs, schedules and retention: [Scan engine](scan-engine.md). - What the pages show: [Web UI](web-ui.md). - -## Open questions - -1. Docker Hub references are not normalized to one form (`docker.io/nginx` vs `nginx`, `index.docker.io` vs - `docker.io`), so one image can have several catalog entries and not be recognized as shared. Should the parser - unify them? -2. Manual registration matches an existing observed image by the untrimmed, case-sensitive name and names are not - unique, so near-identical names create separate entries, and registering a manual image under a discovered image's - name updates the discovered one (which the next discovery run overwrites). Is name the intended identity? -3. Observed images can neither be disabled nor deleted by the user, and discovery re-enables every listed repository on - each run. Is a user-controlled enabled state planned? -4. The recorded "registered by" kind of an image version is last-writer-wins and a further kind (registry metadata) is - defined but never written. Is the field meant to carry meaning? diff --git a/docs/areas/update-detection.md b/docs/areas/update-detection.md index 2af0303..7287a4a 100644 --- a/docs/areas/update-detection.md +++ b/docs/areas/update-detection.md @@ -138,7 +138,9 @@ scan run *partial*, with the message added to the run ([Scan engine](scan-engine ([Image catalog](image-catalog.md)). - The candidates of the evaluation are stored as tag candidates with tag, digest, publication time, the evaluation summary as reason, a rank equal to the candidate's 1-based position in the evaluation and a *recommended* flag on the - candidate whose tag equals the recommended tag. Candidates without a digest are not stored, so ranks may have gaps. + candidate whose tag equals the recommended tag. Candidates without a digest are not stored, since they offer no + reliable update target; a finding can therefore hold fewer candidates than the evaluation produced, and ranks may + have gaps. A finding holds each tag/digest pair at most once. - Whenever a container was evaluated (including *no tag data* and *unsupported*), all active findings of the same container on the same Docker instance are first deactivated: they become inactive, get a resolution timestamp and @@ -168,6 +170,16 @@ scan run *partial*, with the message added to the run ([Scan engine](scan-engine - Evaluation itself performs no I/O and has no failure state of its own; registry failures reach it only as missing tags and are reported through the assessment status above, with the registry's failure message. +### Known gaps + +- The manual tag selection is informational only: it does not pin the recommendation or influence any container + action. Tracked in [#291](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/291). +- Findings of type *base image update* are defined in the data model but never produced. Tracked in + [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). +- The major-upgrade age conditions read the system clock directly instead of the application's time source, so they + cannot be controlled independently of the real time. + Tracked in [#289](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/289). + ## Related decisions None yet. @@ -181,12 +193,3 @@ None yet. - Container discovery and snapshots: [Docker instances](docker-instances.md). - Applying an update through Portainer: [Container actions](container-actions.md). - How findings, candidates and the selection are displayed: [Web UI](web-ui.md). - -## Open questions - -1. The manual tag selection is stored and displayed but has no effect on update evaluation or on container actions. - Is it meant to stay informational, or should it pin the recommendation or feed the Portainer update action? -2. Tag candidates without a digest are dropped when a finding is stored, which can leave a *runtime image update* - finding with fewer candidates than the evaluation produced (or none) and ranks with gaps. Is that intended? -3. Findings of type *base image update* are defined but never created. Is the type reserved for future use or - obsolete? From 6ed6b5a797f22e61062353cb8bbad13e989c4397 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 20:09:22 +0000 Subject: [PATCH 11/12] Resolve open questions of the remaining areas (in progress) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- docs/areas/container-actions.md | 33 +++++++++++---------- docs/areas/docker-instances.md | 33 ++++++++++----------- docs/areas/hosting.md | 38 +++++++++++++----------- docs/areas/registries.md | 51 +++++++++++++++++---------------- docs/areas/scan-engine.md | 26 ++++++++--------- docs/areas/telemetry.md | 32 ++++++++++----------- docs/areas/vulnerabilities.md | 33 +++++++++++---------- docs/areas/web-ui.md | 28 +++++++++--------- 8 files changed, 140 insertions(+), 134 deletions(-) diff --git a/docs/areas/container-actions.md b/docs/areas/container-actions.md index c4bf942..2b2118b 100644 --- a/docs/areas/container-actions.md +++ b/docs/areas/container-actions.md @@ -77,7 +77,7 @@ transport security; the UI is the intended caller of the actions. - A container snapshot referenced by an action run is never removed by retention ([Scan engine](scan-engine.md)). Deleting the Portainer endpoint or the snapshot keeps the action run and clears the link. - Today no component executes actions or writes action runs: neither a page nor a background job calls the capability - check or an action (see *Open questions*). + check or an action (see *Known gaps*). ## Limits and errors @@ -88,6 +88,21 @@ transport security; the UI is the intended caller of the actions. - Error messages and log entries name the Docker instance, the Portainer base URL, the action, the container name and HTTP status codes. They never contain the password, the API token or the JWT. +### Known gaps + +- Container actions cannot be triggered from the UI or a job, and no action run is ever recorded. The executable + actions (`start`, `stop`, `restart`, `kill`, `pause`, `unpause` on containers) do not match the action types of the + action-run record, and redeploying stacks and updating services are not implemented. Tracked in + [#280](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/280). +- The login and the action request are retried on transient answers like any other request, so one requested action + can be executed more than once ([Registries](registries.md)). Tracked in + [#273](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/273). +- `AllowInsecureHttp: true` logs no runtime warning, unlike `SkipCertificateValidation` for Docker instances. Tracked + in [#276](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/276). +- Without `EndpointId` the first environment Portainer lists is used, which can be an environment other than the one + that manages the Docker instance; an explicit `EndpointId` is not required when Portainer manages several + environments. Tracked in [#279](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/279). + ## Related decisions None yet. @@ -99,19 +114,3 @@ None yet. - Startup validation failure and configuration reload: [Hosting](hosting.md). - Retention of snapshots and other history: [Scan engine](scan-engine.md). - What the UI shows about Portainer availability: [Web UI](web-ui.md). - -## Open questions - -1. No page or job triggers Portainer actions, and no action run is ever written. Is the integration meant to be wired - into the UI (the security area list names "the UI that triggers them"), or should the unused parts be removed? -2. The executable actions (`start`, `stop`, `restart`, `kill`, `pause`, `unpause` on containers) do not match the - action types of the action-run record (*restart container*, *redeploy stack*, *update service* on containers, - services and stacks). Which set is the intended contract? Redeploying stacks and updating services are not - implemented. -3. The action request and the login are retried on HTTP 408, 429 and 5xx answers like any idempotent request, so one - requested action can be executed more than once. Should non-idempotent Portainer requests be excluded from retry? -4. `AllowInsecureHttp` produces no runtime warning, while the documentation of the security posture says it is logged - as a warning at runtime. Should a warning be added, as for `SkipCertificateValidation`? -5. Without `EndpointId`, the first environment Portainer lists is used, which can be an environment other than the one - that manages the Docker instance. Should an explicit `EndpointId` be required when Portainer manages more than one - environment? diff --git a/docs/areas/docker-instances.md b/docs/areas/docker-instances.md index 0480766..f1d9314 100644 --- a/docs/areas/docker-instances.md +++ b/docs/areas/docker-instances.md @@ -25,9 +25,10 @@ on the snapshots and samples it produces. - creates a record for each new name, with registration source *configuration file*; - updates endpoint URI (trimmed `BaseUrl`), connection kind, enabled flag and `SkipCertificateValidation` of each existing one; - - **deletes** every stored instance whose name is no longer configured, together with its scan runs, container - snapshots, resource samples, Portainer endpoint and action runs. Update findings that pointed at a deleted - snapshot lose that link but are kept. + - **deletes** every stored instance whose name is no longer configured, immediately and without waiting for + retention, together with its scan runs, container snapshots, resource samples, Portainer endpoint and action + runs. A removed instance is not kept as disabled. Update findings that pointed at a deleted snapshot lose that + link but are kept. - A disabled instance (`Enabled: false`) keeps its stored history but is not contacted: no discovery, no inspection, no resource sampling. Every Engine call for it answers *not configured*. - The stored connection kind is derived from `BaseUrl`: `http` → Http, `https` → Https, `tcp` → Https when `UseTls` @@ -152,6 +153,19 @@ on the snapshots and samples it produces. - `SkipCertificateValidation` and a mounted Docker socket are trust relaxations the operator chooses; the socket is root-equivalent on the host ([Hosting](hosting.md)). +### Known gaps + +- Instance lookups during a runtime scan or a resource collection compare the configured `Name` untrimmed with the + trimmed stored name, so an instance whose configured name has leading or trailing spaces is stored but always + skipped as missing from configuration. Tracked in + [#266](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/266). +- A `CertificatePath` that is not a `.pfx`/`.p12` file is loaded as a certificate without a private key, which cannot + authenticate a TLS client, and a `CertificatePath` file that does not exist is ignored without validation error or + warning. Tracked in [#269](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/269). +- A single timed-out or broken stats request discards the whole round of samples for its instance, while a + non-success stats answer only drops that container. Tracked in + [#270](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/270). + ## Related decisions None yet. @@ -165,16 +179,3 @@ None yet. - Restart and other actions through Portainer: [Container actions](container-actions.md). - Pages that show instances, containers and resource charts: [Web UI](web-ui.md). - Startup, validation failure and the container image's socket access: [Hosting](hosting.md). - -## Open questions - -1. Instance lookups during a scan or a resource collection match the configured `Name` untrimmed, while the stored - name is trimmed. A configured name with leading or trailing spaces is therefore stored but always skipped as - "missing from configuration". Is that intended, or should both sides trim? -2. A `CertificatePath` that is not a `.pfx`/`.p12` file is loaded as a certificate without a private key, which - cannot authenticate a TLS client. Should PEM certificate-plus-key files be supported, or only PKCS#12 be accepted? -3. A missing `CertificatePath` file is ignored silently. Should it fail validation or log a warning? -4. A single stats timeout discards the whole round of samples for an instance, while a non-success stats answer only - drops that container. Is the asymmetry intended? -5. Removing an instance from configuration deletes its whole history, including scan runs, without retention. Is - that the desired behavior, or should a removed instance be kept as disabled? diff --git a/docs/areas/hosting.md b/docs/areas/hosting.md index b520feb..7994ea9 100644 --- a/docs/areas/hosting.md +++ b/docs/areas/hosting.md @@ -85,7 +85,8 @@ registry never delays or prevents startup. - `DockerUpdateGuard:DockerHub:ApiBaseUrl`, `DockerUpdateGuard:DockerHub:RequestTimeoutSeconds` (also the request timeout for generic registries and release-metadata feeds) and `DockerUpdateGuard:ReleaseMetadata:*`; - the whole `Telemetry` section. -- A reloaded configuration is validated by the same rules as at startup when it is next read; see *Open questions*. +- A reloaded configuration is validated by the same rules as at startup when it is next read; a reload that fails + validation raises an error where it is read (see *Known gaps*). ### HTTP pipeline @@ -153,6 +154,26 @@ These properties hold today and must be provided for by the deployment: | Unhandled exception in a request (non-Development) | The `/error` page with a generic message. | | Unknown URL or bodyless error status | The `/not-found` page. | +### Known gaps + +- An invalid configuration reload is not rejected while the last valid configuration stays in use: the background + jobs read their interval from the reloaded configuration outside their error handling, so a reload that fails + validation can stop the host ([Scan engine](scan-engine.md)). + Tracked in [#281](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/281). +- Forwarded headers from a reverse proxy are not supported. Behind a TLS-terminating proxy the application sees + plain HTTP, so it never sends HSTS and HTTPS redirection cannot determine a port; both are left to the proxy. + Tracked in [#283](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/283). +- Certificates in `/app/certs` are not imported for the default user: in the shipped image the system trust store is + owned by root and not writable by group `0`, so the import is skipped unless the container runs as root. + Tracked in [#285](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/285). +- Waiting for the migration advisory lock is bounded by the 30-second database command timeout and then retried like + any transient database failure (see *Database connection*), so an instance that starts while another one runs a + long migration can fail its startup after several minutes. + Tracked in [#287](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/287). +- The shipped `appsettings.json` configures one Docker instance with an empty name, which fails configuration + validation; the application does not start with the shipped settings until that instance is named or removed. + Tracked in [#288](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/288). + ## Related decisions None yet. @@ -168,18 +189,3 @@ None yet. - Pages and what they show: [Web UI](web-ui.md). - Building, testing, CI and releasing the image: `docs/CONTRIBUTING.md`. - Compose files and the walkthrough for the deployment tiers: `INSTALL.md`. - -## Open questions - -1. Background jobs read their interval from the reloaded configuration outside their error handling. If a reload - produces a configuration that fails validation, reading it raises an error there that is not caught, which can - stop the host instead of keeping the last valid configuration. Is stopping the intended reaction to an invalid - reload? -2. There is no handling of forwarded headers from a reverse proxy. Behind a TLS-terminating proxy the application - sees plain HTTP, so HSTS is never sent and HTTPS redirection cannot determine a port. Is relying on the proxy for - both intended? -3. In the shipped image the trust store is owned by root and not writable by group `0`, so the certificate import - from `/app/certs` appears to be skipped unless the container is started as root. Should the import work for the - default user? -4. An instance waiting for the migration advisory lock is subject to the database command timeout. Whether a long - migration by one instance can make a concurrently starting instance fail its startup is not established. diff --git a/docs/areas/registries.md b/docs/areas/registries.md index 51dd9af..72dbaa4 100644 --- a/docs/areas/registries.md +++ b/docs/areas/registries.md @@ -113,9 +113,11 @@ retry policy and the failure model. - `Scanning:DockerHubRequestLimitWindowHours` (default 6, range 1–168), `Scanning:DockerHubRequestLimitPerWindow` (default 200, range 1–100000) and `Scanning:DockerHubReservedManualRequestsPerWindow` (default 40, range 0–100000, - smaller than the per-window limit) describe the Docker Hub request budget. They only stretch the schedule of the - observed-image scan ([Scan engine](scan-engine.md)). Requests are not counted, manual scans are not limited, and no - request is refused for budget reasons. + smaller than the per-window limit) describe the Docker Hub request budget. The budget is a scheduling hint: it + assumes one Docker Hub request per observed image and scan and only stretches the schedule of the observed-image + scan ([Scan engine](scan-engine.md)). It does not count requests; a scan may issue several requests per image + (token, manifests, configuration blobs, tag lookups and listings), manual scans are not limited, and no request is + refused for budget reasons. - `DockerUpdateGuard:DockerHub:MaxParallelRequests` (default 4, range 1–32) caps the requests one Docker Hub client instance has in flight at once, covering web API, login, token and registry requests. Every scan and every UI operation works with its own client instances, so the cap is not global. The value is read when an instance is @@ -167,6 +169,27 @@ retry policy and the failure model. - Registry responses are untrusted: missing or mistyped JSON fields are treated as unknown values. An unbounded listing (account discovery) reads every page the registry offers. +### Known gaps + +- `MaxParallelRequests` caps each Docker Hub client instance, not the process: concurrent scans, and the client + instances one scan uses, each get the full allowance. Tracked in + [#265](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/265). +- Non-idempotent requests with a body, such as the Docker Hub login and Portainer actions, are retried like any other + request, so one request can take effect more than once. Tracked in + [#273](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/273). +- `DockerHub:Registry` does not decide which registries the Docker Hub client serves (that host list is fixed); it + only gates the account listing and, when set to an absolute URL with a non-Docker-Hub host, replaces the web API + base address. Tracked in [#268](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/268). +- Transport errors, timeouts and malformed JSON escape as errors from several calls instead of being answered as + *failed* outcomes. Tracked in [#272](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/272). +- For other registries an image configuration lookup without a platform takes the first index entry, which can be an + attestation manifest, and publication times come only from a top-level `created` field of the manifest, which most + registries do not send, so their tags usually have no publication time. Tracked in + [#277](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/277). +- Pagination links (`next` on Docker Hub, `Link` elsewhere) are followed to whatever host they name, and the Docker + Hub bearer token is sent along to that host. Tracked in + [#264](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/264). + ## Related decisions None yet. @@ -180,25 +203,3 @@ None yet. - Version lines, tag ranking and update recommendations: [Update detection](update-detection.md). - Docker Scout and Trivy: [Vulnerabilities](vulnerabilities.md). - Docker Engine access: [Docker instances](docker-instances.md); Portainer: [Container actions](container-actions.md). - -## Open questions - -1. `MaxParallelRequests` limits each Docker Hub client instance, not the process: concurrent scans, and the two client - instances one scan resolves, each get the full allowance. Is a process-wide cap intended? -2. The request budget assumes one Docker Hub request per observed image and scan, while a scan issues several (token, - manifest, platform manifest and blob per chain level, tag lookups and up to 150 tags per floating base tag). Should - the budget count real requests, or be documented as a rough scheduling hint only? -3. Requests with a body are retried like any other, including the Docker Hub login (and Portainer actions, see - [Container actions](container-actions.md)). Should non-idempotent requests be excluded? -4. `DockerHub:Registry` does not decide which registries the Docker Hub client serves (that host list is fixed). It - only gates the account listing and, when set to an absolute URL with a non-Docker-Hub host, replaces the web API - base address. What is the intended meaning of the key? -5. Transport errors, timeouts and malformed JSON escape as errors from several calls (see *Limits and errors*), which - contradicts the rule that integrations answer with outcomes. Should they map to *failed*? -6. For other registries an image configuration lookup without a platform takes the first index entry, which can be an - attestation manifest; Docker Hub skips `unknown` platforms. Should both behave alike? -7. Publication times for other registries come only from a top-level `created` field of the manifest, which most - registries do not send, so those tags usually have no publication time. Should the configuration blob's `created` - be used instead? -8. Pagination links (`next` on Docker Hub, `Link` elsewhere) are followed to whatever host they name, and the Docker - Hub bearer token is sent along. Should pagination be restricted to the original host? diff --git a/docs/areas/scan-engine.md b/docs/areas/scan-engine.md index 08ff394..7320191 100644 --- a/docs/areas/scan-engine.md +++ b/docs/areas/scan-engine.md @@ -184,6 +184,19 @@ may scan the same image at the same time. This is wasteful but does not corrupt - If the database cannot be reached during a cleanup pass, the whole pass fails, nothing is removed, and the next pass starts after one interval. +### Known gaps + +- Reading a job's interval is outside the job's error handling. When it fails — a configuration reload with an + invalid value, or an unreachable database while the observed-image refresh counts the enabled images — the job + ends with an error and the host stops, instead of the job keeping its last known interval. + Tracked in [#281](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/281). +- The 20 most recently completed scan runs are a cap, not a minimum: an unreferenced run completed before the + `RetainScanRunsDays` cutoff is removed even when it is among the 20 most recent, so the scan history can be + empty after a long pause. Tracked in [#290](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/290). +- Runtime container scans and vulnerability refreshes have no per-subject guard: a manual and a scheduled run of the + same kind can run at the same time (see *Overlapping scans*). + Tracked in [#292](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/292). + ## Related decisions None yet. @@ -200,16 +213,3 @@ None yet. - How scan runs and manual triggers are shown in the UI in [Web UI](web-ui.md). - Metrics recorded per run and the inventory gauges in [Telemetry](telemetry.md). - Configuration loading, validation and the startup sequence in [Hosting and configuration](hosting.md). - -## Open questions - -1. The interval of a job is read outside its error handling. If reading it fails — the observed-image refresh counts - enabled images in the database for every interval, and a configuration reload with an invalid value makes the - options unreadable — the job ends with an error, and with the platform's default behavior the host stops. Is a - failing interval lookup meant to stop the host, or should the job fall back to the last known interval? -2. Scan-run retention keeps unreferenced completed runs only while they are both among the 20 most recent and - within `RetainScanRunsDays`, i.e. 20 is a cap, not a minimum kept past retention. Is the cap the intended - behavior (the configuration reference describes `RetainScanRunsDays` as the retention period of the scan - history)? -3. Runtime container scans and vulnerability refreshes can overlap between a manual and a scheduled run, unlike - observed-image scans. Is overlap acceptable for them, or should they get the same per-subject guard? diff --git a/docs/areas/telemetry.md b/docs/areas/telemetry.md index 678dd30..c65bef8 100644 --- a/docs/areas/telemetry.md +++ b/docs/areas/telemetry.md @@ -51,7 +51,7 @@ Every exported signal carries the same resource: | `dockerengine.request` | Container discovery on one Docker instance | `docker.instance.name` (the configured instance name) | - The span names `scan.run`, `cve.request`, `portainer.request`, `portainer.action` and `persistence.operation` are - reserved for the application's source but not emitted (see *Open questions*). + reserved for the application's source but not emitted (see *Known gaps*). ### Logs @@ -75,8 +75,8 @@ Every exported signal carries the same resource: | `dockerupdateguard.scans.duration` | Histogram | `s` | The run's duration in seconds | All three carry `scan.type` (`ObservedImage`, `RuntimeContainer` or `Vulnerability`) and `result.status` (the - final status, for example `Succeeded`, `Partial` or `Failed`). A run that ends `Partial` is counted in - `scans.total` but not in `scans.failed`. + final status, for example `Succeeded`, `Partial` or `Failed`). `scans.failed` counts only runs that produced no + usable result; a run that ends `Partial` is not a failed run and is counted in `scans.total` only. - Inventory gauges: | Name | Value | @@ -115,6 +115,18 @@ Every exported signal carries the same resource: database is contacted. - Export failures are handled inside the exporter and are never surfaced to users or scans. +### Known gaps + +- Scans, vulnerability provider calls, Portainer calls and persistence have no application span; the reserved span + names `scan.run`, `cve.request`, `portainer.request`, `portainer.action` and `persistence.operation` are never + emitted. Tracked in [#83](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/83). +- `dockerupdateguard.runtime_containers` counts every container that still has a stored snapshot, including + containers that have since stopped or been removed, until cleanup deletes their snapshots; it does not count only + the currently known containers. Tracked in [#294](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/294). +- Configured base URLs (Docker instance, Portainer) are logged verbatim, and retry log lines keep the user-info part + of a URL, so credentials embedded in a URL (`https://user:secret@host`) reach the logs. + Tracked in [#295](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/295). + ## Related decisions None yet. @@ -128,17 +140,3 @@ None yet. - Secrets in the UI: [Web UI](web-ui.md); secrets in configuration: [Hosting](hosting.md). - Runtime warnings for `SkipCertificateValidation` and `AllowInsecureHttp`: [Docker instances](docker-instances.md) and [Container actions](container-actions.md). - -## Open questions - -1. The span names `scan.run`, `cve.request`, `portainer.request`, `portainer.action` and `persistence.operation` are - defined but no code starts such spans, so scans, vulnerability provider calls, Portainer calls and persistence - have no application span. Should they be emitted, or the names dropped? -2. `dockerupdateguard.runtime_containers` counts every container that still has a stored snapshot, including - containers that have since stopped or been removed, until cleanup deletes their snapshots. Is "containers seen - within the retention period" the intended meaning, or should it count only currently known containers? -3. Configured base URLs (Docker instance, Portainer) are logged verbatim, and the retry log keeps the user-info part of - a URL. A base URL with embedded credentials (`https://user:secret@host`) would therefore reach the logs. Should - URLs be stripped of user info before logging? -4. A scan run that ends `Partial` is not counted in `dockerupdateguard.scans.failed`. Is that the intended alerting - signal, or should partial runs be counted separately? diff --git a/docs/areas/vulnerabilities.md b/docs/areas/vulnerabilities.md index fb6548a..2fe9932 100644 --- a/docs/areas/vulnerabilities.md +++ b/docs/areas/vulnerabilities.md @@ -151,6 +151,22 @@ The dashboard shows at most one hint about the vulnerability configuration: - Messages are built from the image reference and the provider's error text; no credentials or tokens are put into them. +### Known gaps + +- Docker Scout follows the `next` link of a response to whatever host it names and sends the bearer token along; the + same gap in the registry clients is owned by [Registries](registries.md). Tracked in + [#264](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/264). +- Docker Scout requests do not go through the transient-failure retry the other outbound HTTP calls use, and Docker + Scout scans the tag, not the digest, so an image version whose tag was republished is assessed with the advisories + of the newer image. Tracked in [#267](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/267). +- Provider values longer than the stored lengths (for example a Trivy title over 300 characters) are not shortened; + saving them fails and ends the whole refresh *failed*, leaving the remaining image versions unscanned. Tracked in + [#271](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/271). +- Some values are kept although nothing produces them: validation makes the *Trivy base URL missing* hint and the + Trivy *not configured* result unreachable (unless an invalid reloaded configuration is still served), and the + finding sources *registry advisory* and *manual* exist in the data model but are never produced. Tracked in + [#293](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/293). + ## Related decisions None yet. @@ -163,20 +179,3 @@ None yet. - Validation at startup, the bundled Trivy CLI in the container image: [Hosting](hosting.md). - How assessments, findings, severities and the rescan control are displayed: [Web UI](web-ui.md). - Inventory gauges for active findings: [Telemetry](telemetry.md). - -## Open questions - -1. Docker Scout follows the `next` link of a response as given and sends the bearer token with it, without checking - that it points to `DockerScoutBaseUrl`. Should the link be restricted to the configured host? -2. Docker Scout requests do not go through the transient-failure retry that the other outbound HTTP calls use. Is that - intended? -3. Docker Scout scans the tag, not the digest, so an image version whose tag was republished is assessed with the - advisories of the newer image. Is that acceptable? -4. Provider values longer than the stored lengths (for example a Trivy title over 300 characters) are not shortened; - saving them fails and ends the whole refresh *failed*, leaving the remaining image versions unscanned. Should they - be truncated? -5. Validation rejects `Enabled` with `Provider` `Trivy` and no `TrivyBaseUrl`, so the *Trivy base URL missing* hint - and the Trivy *not configured* result look unreachable (unless an invalid reloaded configuration is still served). - Are they kept on purpose? -6. The finding sources *registry advisory* and *manual* exist in the data model but are never produced. Reserved or - obsolete? diff --git a/docs/areas/web-ui.md b/docs/areas/web-ui.md index b27c356..c95b674 100644 --- a/docs/areas/web-ui.md +++ b/docs/areas/web-ui.md @@ -115,7 +115,7 @@ never their values. - The UI never shows configured secrets: the Docker Hub token, Portainer credentials and token and the database connection string are neither displayed nor sent to the browser. Pages show only whether an integration is configured, and the Docker Hub user name as the title of the My Images list. -- Docker instance endpoints are shown as configured. +- Docker instance endpoints are shown as configured (see *Known gaps*). ## Limits and errors @@ -131,6 +131,20 @@ never their values. - Lists are not paged except the Vulnerabilities page; all other lists show every item or the fixed number stated above. +### Known gaps + +- The scan status chips (dashboard, scan history, observed image and My Images lists) color only `Failed`, `Running` + and `Pending`; `Succeeded` and `Partial` are shown without color, so a partial run is not marked as a warning. + Tracked in [#274](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/274). +- A detail route with a well-formed identifier that does not exist (for example a deleted observed image) keeps + showing the loading skeleton instead of a not-found state. + Tracked in [#40](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/40). +- Docker instance endpoints are not masked: an endpoint that embeds credentials as user information in its URL is + shown with them. Tracked in [#295](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/295). +- The Scan History and dashboard descriptions mention discovery and cleanup runs, although those jobs record no scan + runs ([Scan engine](scan-engine.md)). + Tracked in [#278](https://github.com/LarsLaskowski/DockerUpdateGuard/issues/278). + ## Related decisions None yet. @@ -148,15 +162,3 @@ None yet. - Container actions through Portainer in [Container actions](container-actions.md). - Error and status-code middleware, HTTPS and the missing authentication in [Hosting and configuration](hosting.md). - -## Open questions - -1. The scan status chips (dashboard, scan history, observed image and My Images lists) color `Failed`, `Running` and - `Pending`, but expect `Completed` for success, a status that does not exist; `Succeeded` and `Partial` are shown - without color. Should `Succeeded` be green and `Partial` be marked as a warning? -2. A detail route with a well-formed identifier that does not exist (for example a deleted observed image) keeps - showing the loading skeleton instead of a "not found" state. Is that intended? -3. Docker instance endpoints are shown exactly as configured. Should an endpoint that embeds credentials in its - address (user information in the URL) be masked, or is such configuration unsupported? -4. The Scan History and dashboard descriptions mention discovery and cleanup runs, but those jobs record no scan - runs. Should the text change, or should these jobs record runs? From a71262f236fc8c6939cd9e488dd175111537f34d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 20:32:28 +0000 Subject: [PATCH 12/12] Align the README configuration reference with the current behavior Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01A3Rn7L5t5JApSC3JeXZbNd --- README.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 7fa2e2b..e6721a3 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,7 @@ DockerUpdateGuard uses normal ASP.NET Core configuration binding and prefers con - `appsettings.json` - `appsettings.{Environment}.json` +- environment variables (`__` as the section separator, e.g. `DockerUpdateGuard__Scanning__RetryCount`) - command-line arguments - secret stores supported by ASP.NET Core @@ -96,8 +97,11 @@ For example: | `ConnectionStrings:DockerUpdateGuard` | none | Yes* | Named PostgreSQL connection string used when `DockerUpdateGuard:ConnectionString` is not set | | `DockerUpdateGuard:ConnectionString` | none | Yes* | Inline PostgreSQL connection string | | `DockerUpdateGuard:ConnectionStringName` | `DockerUpdateGuard` | No | Name of the `ConnectionStrings` entry to resolve | -| `DockerUpdateGuard:DisplayVersion` | assembly version | No | Version string shown in the UI footer; the Docker image sets this automatically | - +| `DockerUpdateGuard:DisplayVersion` | assembly version | No | Version string shown in the navigation drawer; the Docker image sets this automatically | +| `DockerUpdateGuard:Database:MaxConnectionRetryCount` | `8` | No | Retries of a transiently failing database command during normal operation | +| `DockerUpdateGuard:Database:MaxConnectionRetryDelaySeconds` | `30` | No | Maximum back-off in seconds between those retries | +| `DockerUpdateGuard:Database:MigrationStartupTimeoutSeconds` | `120` | No | How long startup waits for the database to become reachable | +| `DockerUpdateGuard:Database:MigrationRetryDelaySeconds` | `5` | No | Delay in seconds between connection attempts while startup waits for the database | \* At least one of `DockerUpdateGuard:ConnectionString` or `ConnectionStrings:{ConnectionStringName}` must be configured. @@ -109,7 +113,7 @@ For example: | `ApiBaseUrl` | `https://hub.docker.com/` | No | Docker Hub API base address used for registries served by Docker Hub | | `UserName` | none | No | Docker Hub username for authenticated API requests | | `Pat` | none | No | Docker Hub personal access token | -| `RequestTimeoutSeconds` | `30` | No | Timeout for outbound Docker Hub and OCI registry requests | +| `RequestTimeoutSeconds` | `30` | No | Total time per outbound Docker Hub, OCI registry and .NET/nginx release-feed request, including its retries | | `MaxParallelRequests` | `4` | No | Maximum logical request parallelism for registry operations | ### `DockerUpdateGuard:ReleaseMetadata` @@ -149,7 +153,7 @@ versions. Override them to route the lookups through an internal mirror or proxy | `ResourceStatisticsIntervalMinutes` | `5` | Interval for sampling CPU, memory, and network usage | | `VulnerabilityRefreshIntervalMinutes` | `180` | Interval for refreshing vulnerability information | | `CleanupIntervalMinutes` | `720` | Interval for cleaning old scan data | -| `RetryCount` | `2` | Retry count for transient background failures | +| `RetryCount` | `2` | Retries of a single outbound HTTP request after a transient failure (`0` to `10`); applies to Docker Hub, other registries, the release feeds and Portainer, for scheduled and manual work alike. A failed scan run is not retried before its next scheduled run | | `RetainScanRunsDays` | `30` | Retention period for completed scan history | | `MajorUpgradeMinimumAgeDays` | `30` | Waiting period before a new major version line is recommended; the upgrade also stays deferred while the current major version line is still served next to the new line or while its first release is unknown; `0` disables the waiting periods | | `MajorUpgradeMinimumReleaseCount` | `2` | Number of releases a new major version line must have before it is recommended; `0` disables the requirement | @@ -205,9 +209,9 @@ must be configured. | `ServiceName` | `DockerUpdateGuard` | Required when telemetry is enabled | Service name used in telemetry resources | | `OtlpEndpoint` | none | No | Absolute `http` or `https` OTLP endpoint | | `Instance` | none | No | Logical deployment instance name | -| `EnableLogging` | `true` | No | Enables OpenTelemetry logging export | -| `EnableMetrics` | `true` | No | Enables OpenTelemetry metrics export | -| `EnableTracing` | `true` | No | Enables OpenTelemetry tracing export | +| `EnableLogging` | `true` | No | Enables OpenTelemetry logging (exported only when `OtlpEndpoint` is set; console logging is always active) | +| `EnableMetrics` | `true` | No | Enables OpenTelemetry metrics (exported only when `OtlpEndpoint` is set) | +| `EnableTracing` | `true` | No | Enables OpenTelemetry tracing (exported only when `OtlpEndpoint` is set) | If all three telemetry switches are `false`, telemetry is effectively disabled.