From 52eb4d7ec79bf31a2d13e95a2466e986c211e070 Mon Sep 17 00:00:00 2001 From: Sunil Kumar Date: Mon, 17 Aug 2026 13:26:29 +0000 Subject: [PATCH] docs(machine-validation): address coderabbit feedback --- .../pluggable-machine-validation-framework.md | 536 ++++++++++++++++++ 1 file changed, 536 insertions(+) create mode 100644 designs/machine-validation/pluggable-machine-validation-framework.md diff --git a/designs/machine-validation/pluggable-machine-validation-framework.md b/designs/machine-validation/pluggable-machine-validation-framework.md new file mode 100644 index 0000000000..14ab35d075 --- /dev/null +++ b/designs/machine-validation/pluggable-machine-validation-framework.md @@ -0,0 +1,536 @@ +# Pluggable Machine Validation Framework + +## Software Design Document + +## Revision History + +| Version | Date | Modified By | Description | +| :---: | :---: | :---- | :---- | +| 0.1 | 2026-08-14 | Sunil Kumar | Initial draft | +| | | | | + +# **1. Introduction** + +Machine Validation checks whether a machine is ready to use. Today it runs a predefined set of validation tests. If a site needs a different check, it normally needs a Machine Validation code or catalog change, or it must run the check outside Machine Validation. + +This design adds a supported plugin model. A site administrator can configure a plugin that contains a site-specific validation check. Machine Validation still owns the surrounding work: selecting the check, running it, enforcing its timeout, tracking its status, storing the result, and reporting the final machine outcome. + +The first version supports plugins packaged as container images. It does not define the validation logic inside those containers. + +## **1.1 Purpose** + +The purpose of this document is to give NICo engineers, site administrators, and plugin authors a shared, simple design for site-owned Machine Validation checks. It focuses on: + +1. Allowing an authorized site admin to configure a validation plugin without changing Machine Validation core code. +2. Giving every plugin the same input and output contract. +3. Running plugins safely and consistently through Scout. +4. Keeping Machine Validation responsible for lifecycle, status, timeout, and reporting behavior. + +## **1.2 Definitions and Acronyms** + +| Term/Acronym | Definition | +| :---- | :---- | +| NICo | NVIDIA bare-metal lifecycle management system. | +| SDD | Software Design Document. | +| Scout | Temporary host-side agent that runs during discovery and validation. | +| Machine Validation | Process that validates a machine before or after tenant use. | +| Plugin | A site-provided package that performs one validation check. | +| Plugin Definition | A site-scoped, versioned catalog entry that tells Machine Validation when and how to run a plugin. | +| Verified Revision | An immutable plugin revision that NICo has validated and the site admin has approved. Only verified revisions can be enabled. | +| Run | One Machine Validation request for one machine and context. | +| Run Item | One selected validation check inside a run. | +| Attempt | One execution of a run item. | +| OCI image | A standard container image format. | + +## **1.3 Scope** + +This SDD covers the framework needed to configure and run validation plugins. + +1. Plugin configuration and validation. +2. A standard plugin input file and result file. +3. Container-based plugin execution in Scout. +4. Timeout, status, result, logging, audit, and visibility integration with existing Machine Validation behavior. +5. Safe defaults for permissions and package sources. + +This SDD does not cover: + +1. Creating or maintaining actual validation tests. +2. Plugin-specific business logic. +3. Every possible site-specific validation workflow. +4. Supporting scripts, archives, or remote services in the first release. +5. Replacing the existing Machine Validation lifecycle and orchestration. + +### **1.3.1 Assumptions, Constraints, Dependencies** + +Assumptions: + +1. Scout is the execution environment for Machine Validation plugins. +2. Existing Machine Validation runs, results, and operator workflows continue to work. +3. A plugin can determine its own check result, but it cannot directly change Machine Validation run state. +4. Some verified plugin revisions may need root and direct hardware access to perform machine validation. + +Constraints: + +1. Plugins run unprivileged by default. Only a verified revision may opt into root and privileged container access. +2. A plugin must use a stable, versioned contract. +3. Each run must record the exact plugin version it used. +4. The first release runs selected plugins sequentially; future parallel execution must preserve the same input/output contract and run-item lifecycle. + +Dependencies: + +1. Existing durable run, run-item, and attempt tracking. +2. Scout container runtime and image-pull support. +3. Existing API, database, UI, CLI, and audit mechanisms for Machine Validation. + +## **1.4 Requirements Summary** + +### **1.4.1 Functional Requirements** + +| ID | Requirement | +| :--- | :--- | +| FR-1 | Site admins can create, update, enable, disable, and view plugin definitions through the Machine Validation API. UI and CLI support follow after the MVP. | +| FR-2 | A plugin definition states when the plugin applies, which container image to run, and its timeout and permission settings. | +| FR-3 | Machine Validation selects plugins using existing context, platform, and tag rules. | +| FR-4 | Scout gives every plugin a standard JSON input file. | +| FR-5 | Every plugin writes a versioned, schema-validated JSON result file with `pass`, `fail`, or `error`. | +| FR-6 | Machine Validation records plugin version, execution status, timeout, and result with the run item and attempt. | +| FR-7 | Existing run, result, UI, CLI, and REST workflows show plugin-backed results. | +| FR-8 | A plugin definition change does not affect a plugin that has already been selected for a run. | + +### **1.4.2 Non-Functional Requirements** + +| ID | Requirement | +| :--- | :--- | +| NFR-1 | Plugins run unprivileged by default. Only a verified revision may opt into root and privileged container access. | +| NFR-2 | Plugin packages use an immutable container image reference pinned to a digest. | +| NFR-3 | Plugin output, logs, and diagnostic files are size-limited and redacted before storage or display. | +| NFR-4 | Plugin configuration and execution are auditable. | +| NFR-5 | A plugin timeout, crash, or invalid result cannot leave the Machine Validation run stuck. | + +# **2. System Architecture** + +## **2.1 High-Level Architecture** + +```text +Site admin manages a plugin in the Machine Validation catalog + | + v +Machine Validation selects the plugin for a machine + | + v +Scout downloads and starts the plugin container + | + v +Plugin reads its input file and writes its result file + | + v +Machine Validation stores the result and updates the run +``` + +Machine Validation remains the system of record. The plugin only performs its site-specific check. Scout runs the plugin and reports the outcome; the API owns the durable run state and the machine-controller continues to use that state as it does today. + +**Configuration model:** site configuration sets the security guardrails; the Machine Validation plugin catalog manages individual plugins, their revisions, and their enabled state. + +## **2.2 Component Breakdown** + +| Component | Responsibility | +| :--- | :--- | +| Site admin | Creates, approves, enables, disables, and views site plugins through the Machine Validation catalog API; UI and CLI support follow after the MVP. | +| Site configuration | Defines guardrails: approved registries, resource limits, permitted capabilities, and permitted full-host contexts or machines. | +| Machine Validation API | Stores immutable plugin revisions, applies site-config guardrails, selects plugins, records results, and owns run state. | +| Scout | Downloads, starts, monitors, and stops the plugin container. | +| Plugin | Performs one site-specific validation check and writes a result. | +| Admin UI / CLI / REST | Shows plugin configuration, execution details, and results. | + +# **3. Detailed Design** + +## **3.1 Plugin Definition** + +A plugin definition is the configuration for one site-owned check. It is stored in a site-scoped Machine Validation plugin catalog and is created or changed through the Machine Validation API in the MVP. UI and CLI support follow after the MVP. This lets a site admin add or disable a plugin without a full site-config update. + +Site configuration is still the source of truth for the guardrails that control the catalog. It defines approved registries, permitted capabilities, maximum resource limits, and permitted full-host contexts or machines. A plugin catalog entry cannot override those policies. Full-host access always needs separate approval. The site-admin walkthrough shows an example policy. + +The policy is stored in the NICo API per-site configuration file, `nico-api-site-config.toml`, under the new `[machine_validation_config.plugin_policy]` section. It is not stored in the plugin catalog. In a Kustomize deployment, the site team maintains this file in its environment overlay and delivers it through the `nico-api-site-config-files` ConfigMap. In a Helm deployment, the site team sets `siteConfig.nicoApiSiteConfig`, which creates the same ConfigMap. The site team applies its normal configuration rollout so `nico-api` loads the updated site configuration before the new policy is used. + +Each plugin definition is separate from this TOML file. The YAML shown in the examples is a plugin catalog manifest: the site admin may keep it in the site's configuration repository, then submits it through the Machine Validation API. UI and CLI support send the same request when they are added after the MVP. Before storing a revision, the API derives the caller's authorized site from authentication and rejects a manifest whose `site` does not match it. The API then validates the manifest against the loaded site policy and stores the accepted definition as an immutable revision in the Machine Validation database. Scout never reads the YAML file or the site configuration directly; it receives the frozen revision only after Machine Validation selects it for a run. + +- a name and description; +- the validation contexts, platforms, or tags where it should run; +- a container image reference pinned to an immutable digest; +- a timeout; +- any approved permissions; and +- any site-managed, non-secret configuration files. + +The following fields form the immutable executable revision. Approval applies to this exact set of values: + +| Field | Meaning | +| :--- | :--- | +| `image` | OCI image pinned to a digest. | +| `entrypoint` | Argument array started by Scout; it is never passed through a shell. | +| `timeoutSeconds` | Maximum execution time. | +| `resourceLimits` | Maximum CPU, memory, process count, and output storage. | +| `parameters` | Site-defined, non-secret JSON values. The plugin owns their meaning. | +| `inputFiles` | Site-managed, non-secret files written below `/opt/nico/mv/input`. The framework validates their paths and size; the plugin owns each file's contents and schema. | +| `capabilities` | Named runtime access requested by the plugin. Scout implements only documented capability names; arbitrary runtime flags are not accepted. | +| `order` | Required execution order; selected items sort by `order`, then plugin name. | +| `allowParallel` | `false` by default; when the framework supports parallel execution, marks the plugin eligible for parallel scheduling if the scheduler and resource policy also allow it. | +| `resourceClass` | Added with parallel execution; one of `none`, `gpu`, `host`, or `full-host`, used only to select the scheduler lock. | +| `privileged` | `false` by default; enables the privileged container profile when approved. | +| `hostAccess` | `none` by default; `full` enables the legacy-style writable host-root mount when separately approved. | + +Verification, full-host approval, and enablement are separate server-managed state, not executable fields. The API stores `isVerified: false`, `isFullHostApproved: false`, and `enabled: false` with the exact revision ID and image digest. After policy validation, the site admin approves the revision and the API records `isVerified: true` for that exact revision ID and digest. A `hostAccess: full` revision needs a separate full-host approval, which records `isFullHostApproved: true` for that same revision ID and digest. Only matching verified state—and, for full-host access, matching full-host approval—can be enabled or launched. A new executable revision always starts with all of this state false. + +Example plugin definition for `example-ai-west-prod`: + +```yaml +site: example-ai-west-prod +name: hardware-inventory +description: Check that the machine inventory is available +runWhen: + contexts: [Discovery, OnDemand] + platforms: [HGX-B200] +image: registry.example.com/example-ai-west-prod/hardware-inventory@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef # Illustrative digest +entrypoint: ["/plugin/entrypoint"] +timeoutSeconds: 300 +resourceLimits: + cpuMillis: 2000 + memoryMiB: 1024 + maxProcesses: 256 +capabilities: [host-inventory-read] +order: 200 +allowParallel: false +privileged: false +hostAccess: none +``` + +When a plugin definition is created or updated, the API validates it against the site's guardrails. For example, it rejects a non-approved registry, a moving image tag, an invalid timeout, or an unsupported capability. For `inputFiles`, it rejects absolute paths, NUL bytes, `..` path segments, duplicate normalized paths, `input.json`, and paths that resolve through a symlink. It creates parent directories without following symlinks, limits the combined input-file size to 64 KiB, and does not interpret plugin-specific file contents. Create and update requests must not contain `isVerified`, `isFullHostApproved`, or `enabled`; the API rejects those server-managed fields. Each accepted change creates a new immutable plugin revision and separate unverified, disabled state. After the site admin approves a revision that passes policy checks, the API records verified state for its exact revision ID and digest. Only a verified revision can be enabled; a full-host revision also needs matching full-host approval. A selected run item keeps a snapshot of every executable field, including parameters, input files, capabilities, and the image digest it will use. + +Plugins are selected as Machine Validation run items, alongside existing tests. Existing selection rules determine whether a plugin matches the machine and context. All selected items use a required `order` and are sorted first by `order`, then by name. Existing tests receive a defined migration order so the combined list is deterministic. The selected run item is the source of truth for execution: disabling or deleting a definition stops future selection, but does not change a plugin already selected for a run. + +## **3.2 Plugin Input and Output Contract** + +### **3.2.1 Input** + +All paths in this contract are paths **inside the plugin container**. Before starting it, Scout creates a separate host-side working directory for the attempt and mounts its input and output subdirectories into the container. The host-side location is an implementation detail; every plugin always sees the same paths: + +```text +/opt/nico/mv/input +/opt/nico/mv/output +``` + +Scout creates a JSON input file and any configured site-managed input files in the read-only input directory. The JSON file includes the run ID, machine ID, platform, validation context, plugin revision, deadline, and site-defined non-secret parameters. + +The input does not contain secrets. It contains only the machine information that the plugin is approved to use. + +The input is stored at `/opt/nico/mv/input/input.json`. It contains `apiVersion: "machinevalidation.nvidia.com/v1"` and `kind: "MachineValidationPluginInput"`. The v1 schema rejects unknown framework fields and values outside documented limits. The supplied `parameters` are frozen in the selected revision and are interpreted only by the plugin. Future incompatible changes require a new major API version. + +Every plugin receives the same input shape, regardless of what it validates. Scout also writes each configured input file to the matching relative path below `/opt/nico/mv/input`. `input.json` is reserved for the framework, so `inputFiles` cannot name it or any normalized alias of it. Scout writes an input like this: + +```json +{ + "apiVersion": "machinevalidation.nvidia.com/v1", + "kind": "MachineValidationPluginInput", + "run": { + "id": "machine-validation-run-id", + "itemId": "run-item-id", + "attempt": 1, + "context": "Discovery", + "deadline": "2026-08-15T12:15:00Z" + }, + "machine": { + "id": "machine-id", + "platform": "HGX-B200" + }, + "plugin": { + "name": "hardware-inventory", + "revision": "3" + }, + "parameters": {} +} +``` + +The plugin reads this file and any configured input files; it does not need to depend on undocumented environment variables or a site-specific API. `inputFiles` must not contain secrets. + +### **3.2.2 Output** + +Before it exits, the plugin writes one JSON result file in a known output location. + +```json +{ + "apiVersion": "machinevalidation.nvidia.com/v1", + "kind": "MachineValidationPluginResult", + "outcome": "pass", + "severity": "info", + "summary": "Hardware inventory is available.", + "findings": [ + { "name": "inventory", "message": "Inventory collection completed" } + ] +} +``` + +The allowed outcomes are: + +| Outcome | Meaning | +| :--- | :--- | +| `pass` | The check completed and the machine met its requirement. | +| `fail` | The check completed and the machine did not meet its requirement. | +| `error` | The plugin could not complete its own work. | + +`severity` is optional context for operators. Its allowed values are `info`, `warning`, `critical`, and `unknown`; it does not change the lifecycle decision, which is based only on `outcome`. This preserves a completed plugin's advisory state without treating it as a failure. + +If the result is missing or invalid, or if the plugin times out or is cancelled, Machine Validation records a framework failure. It does not report it as a normal machine validation failure. + +The result is written to `/opt/nico/mv/output/result.json` using an atomic rename. It contains `apiVersion: "machinevalidation.nvidia.com/v1"` and `kind: "MachineValidationPluginResult"`. Scout validates the schema before accepting it. The result file is limited to 64 KiB; a summary is limited to 4 KiB; findings are limited to 100 entries; and diagnostic output is limited to 1 MiB per attempt. The first release does not accept arbitrary output-file paths or symlinks. + +The shared contract divides responsibility clearly: + +| Machine Validation and Scout provide | Every plugin must provide | +| :--- | :--- | +| A versioned input file at `/opt/nico/mv/input/input.json` | Read the input file and perform its site-specific check. | +| Run identity, machine context, deadline, and approved parameters | Write one valid result file at `/opt/nico/mv/output/result.json`. | +| Container lifecycle, timeout, cancellation, result validation, and persistence | Return `pass`, `fail`, or `error` with a short operator-facing summary. | +| Consistent run/attempt status, logs, audit history, and reporting | Keep output within the documented size limits and avoid writing secrets. | + +This means a storage plugin, GPU plugin, or network plugin is different only in its own validation logic and `parameters`; each uses the same files, result states, and lifecycle. + +| Process outcome | Result file | Machine Validation outcome | +| :--- | :--- | :--- | +| Normal process completion | Valid `pass`, `fail`, or `error` result | Use the result's declared outcome. | +| Normal process completion | Missing, malformed, or inconsistent result | Framework failure: invalid plugin result. | +| Signal, container-runtime error, or other abnormal exit | Any result | Framework failure: ignore the result, even if it says `pass`. | +| Timeout or cancellation | Any result | Framework timeout or cancellation; ignore a late result. | + +### **3.2.3 Plugin Adapter Pattern** + +The framework does not parse tool-specific stdout, exit codes, schemas, or configuration formats. Every plugin image must provide the standard result file. When an existing tool does not already do that, its plugin package includes a small wrapper or adapter that translates between the tool and the framework contract. + +The adapter is part of the plugin image, not Machine Validation core. Scout invokes only the configured entrypoint, mounts the standard directories, applies the approved capability profile, and reads the standard result file. A later plugin can use a different internal tool or wrapper, but Machine Validation does not change. + +## **3.3 Scout Execution Design** + +The first supported package format is an OCI container image. Scout starts the attempt deadline before downloading the exact image recorded in the plugin definition, so `timeoutSeconds` covers both image acquisition and container execution. In the MVP, Scout makes one image-pull attempt; it cancels an in-progress pull when the run is cancelled or its deadline expires. A stalled or failed pull is a terminal framework failure, and retry policy creates a new attempt rather than extending the original deadline. Scout starts the container without a shell. Plugin execution is a new path and must not reuse the current legacy command runner or its privileged host-root mount. + +The plugin receives: + +- a read-only input directory; +- a small writable output directory; and +- only the device, host, or inventory access explicitly approved by policy. + +Plugins run unprivileged with networking disabled by default. Scout enforces this profile as a fixed non-root UID, no Linux capabilities, `no-new-privileges`, and no network connection; an image-declared root user cannot override it. This framework does not support a network-access exception yet. A plugin definition can request `privileged: true` when it needs direct access to hardware devices or low-level system interfaces. The site admin approves that exact revision before it can run. + +When the verified revision has `privileged: true`, Scout starts it as root in the privileged container profile. Privilege does not enable networking or automatically add a writable host-root mount, host PID namespace, or container-runtime socket. Those remain forbidden unless the revision also requests the separate `hostAccess: full` mode. Scout still enforces CPU, memory, process-count, output-storage, and timeout limits, and stops only the container/process group for that attempt. + +The initial capability list is intentionally small: + +| Capability | Access granted | Additional restriction | +| :--- | :--- | :--- | +| `gpu-read` | All GPU devices and the host NVML library through the NVIDIA Container Toolkit | Read-only GPU-health access; no writable host-root mount. | +| `host-journal-read` | Read-only `/run/log/journal`, `/var/log/journal`, and `/etc/machine-id` mounts | Lets a plugin inspect host system events; unavailable paths must produce a visible plugin result, not a silent pass. | +| `host-inventory-read` | A read-only, filtered inventory snapshot | Available to verified revisions; no writable host filesystem access. | + +`privileged` and `hostAccess` are dedicated permission fields, not capabilities. The API rejects capability names that duplicate them, raw container runtime flags, mount paths, or environment variables. The verified revision records the approved runtime profile, and each run snapshots that profile with the image digest. + +### **3.3.1 Privileged Hardware Operations** + +A plugin does not need a separate approval for every privileged hardware operation. When a specific image digest is verified with `privileged: true`, it can run as root in the privileged container profile and use the hardware interfaces available to that profile. + +Verification is granted per plugin revision, never by image name or mutable tag. It requires the image to come from an approved registry and be approved by the site admin. Changing the image digest, entrypoint, input files, capabilities, or privileged setting creates a new revision that must be verified again. + +This model deliberately trades finer-grained runtime controls for a fast operational path for trusted site validation code. The API still prevents the plugin from directly changing Machine Validation state, and it records the site-admin approval, image digest, runtime profile, and execution history for every run. + +### **3.3.2 Full Host Access** + +Some existing Machine Validation tools need the same access as the current legacy runner: a privileged container with the host root filesystem mounted read-write. A plugin revision can request this with `hostAccess: full`. + +`hostAccess: full` requires `privileged: true`, is never the default, and always requires separate full-host approval of the exact revision. The API records that approval separately from ordinary verification, bound to the revision ID and image digest. It requires both states before enablement and again before every launch. Site policy can further limit the allowed contexts or machines, but cannot waive that approval. When enabled, Scout runs the plugin with a read-write bind mount of the host root at `/host`, matching the legacy runner's host access. Scout does not set the host PID namespace or separately pass a container-runtime socket, but this is not a security boundary: the writable host root gives the plugin effective root control of the validation host and may expose host sockets through the mounted filesystem. It is therefore appropriate only for trusted, site-owned validation code and not for third-party or experimental plugins. + +This mode has no technical host isolation. Its controls are the immutable image digest, approval, audit record, and emergency disable; resource limits are best-effort only. A change to `hostAccess`, the image digest, entrypoint, or runtime profile creates a new revision and requires approval again. If cancellation cannot stop host processes created by the plugin, Machine Validation fails the run and requires operator recovery: quarantine and reboot the machine before returning it to service. + +## **3.4 Result and Run Lifecycle** + +1. Machine Validation selects an enabled plugin for the current machine and context. +2. The selected run item records the plugin definition revision and image digest. +3. Scout starts an attempt, sends heartbeats, and runs the container. +4. Scout validates the plugin result and sends it to the API. +5. The API stores the result and updates the existing run and machine status. + +The plugin never calls the API to update its own run. This prevents a plugin from bypassing timeout, status, and audit rules. + +| Plugin/framework outcome | Persisted result and attempt state | Machine Validation behavior | +| :--- | :--- | :--- | +| `pass` | Store the plugin result with `exit_code: 0`; mark the attempt and run item Succeeded. | Continue with the next selected item. | +| `fail` | Store the plugin result with a non-zero validation `exit_code`; mark the attempt and run item Failed. | In `Discovery` and `Cleanup`, fail the run and block the machine lifecycle. In `OnDemand`, record the failed run without changing the machine lifecycle. | +| Plugin `error` | Store the plugin result with a non-zero framework `exit_code` and failure kind `plugin-error`; mark the attempt and run item Failed. | Record an execution error separately from a validation failure. It blocks `Discovery` and `Cleanup`; retry only if policy allows it. | +| Timeout, crash, image-pull failure, or invalid result | Store the failure kind; mark the attempt TimedOut for a timeout, otherwise Failed. A crash or abnormal exit overrides and discards any result file. | Treat it as a framework failure. It blocks `Discovery` and `Cleanup`; reconciliation terminalizes it and applies retry policy. | +| Cancellation | Mark the active attempt and run item Canceled; do not accept a result. | Stop only the active attempt and terminalize the run through the existing control workflow. | + +The API exposes the existing run-item state and result `exit_code`, plus plugin name, revision, image digest, outcome, and failure kind where applicable. A retry creates a new attempt and preserves prior attempt history. Scout ignores a result received after timeout or cancellation; it cannot change a terminal attempt. The final machine-validation status is derived from the terminal run items using the same status mapping as existing tests. + +The existing durable attempt, heartbeat, and reconciler design remains mandatory. A stale plugin attempt is terminalized by the same policy as any other Machine Validation attempt; it cannot leave a machine in validation indefinitely. + +## **3.5 Future Parallel Execution** + +The first release executes plugins one at a time. The input/output contract is designed to support future parallel execution without changing plugin code: every attempt already has its own run-item ID, attempt number, input, result, logs, and status. + +Before enabling parallel execution, the framework adds a frozen `resourceClass` to the executable revision. Its only allowed values are `none`, `gpu`, `host`, and `full-host`; unknown values are rejected. `hostAccess: full` always uses `full-host`, and a plugin that uses a shared GPU or host-wide tool declares `gpu` or `host` respectively. The scheduler must then: + +1. Create a separate host-side working directory for every run item and attempt, then mount its input and output subdirectories into the plugin container as `/opt/nico/mv/input` and `/opt/nico/mv/output`. +2. Use the run-item ID and attempt number for idempotent result, heartbeat, and log updates. +3. Enforce per-machine and per-site maximum parallelism. +4. Run a plugin in parallel only when its frozen revision has `allowParallel: true`. +5. Lock the pair of machine ID and `resourceClass`, then serialize plugins holding the same non-`none` lock. +6. Run only plugins with no ordering dependency at the same time. +7. Define whether a failed plugin allows independent running plugins to finish or cancels them. + +The initial release is sequential. When the framework supports parallel execution, `hostAccess: full` plugins always run alone, `allowParallel: false` plugins always run alone, and plugins with the same resource class run one at a time. Parallel execution is a scheduler change; a plugin still reads `/opt/nico/mv/input/input.json` and writes `/opt/nico/mv/output/result.json` exactly as it does in sequential mode. A non-privileged firmware-inventory plugin with `resourceClass: none` is a typical `allowParallel: true` candidate. + +## **3.6 Admin UI and CLI Design** + +Operators can see the plugin name, version, image digest, attempt status, summary, and allowed diagnostic output alongside existing Machine Validation results. Admins can see why a plugin definition was rejected or disabled. Plugin changes and execution are included in the audit trail. + +## **3.7 Compatibility and Migration** + +Existing built-in validation tests continue to work unchanged through the legacy runner. They continue to use the existing command and result behavior, while sharing the existing run-item lifecycle, timeout handling, status tracking, and reporting. + +New site plugins are created in the site-scoped plugin catalog and selected only for machines in that site. They are selected alongside built-in tests, but use the separate container-plugin runner. Only container plugins receive the standard input and result files. + +Moving a built-in test to a separately packaged plugin is optional and can happen gradually. It is not required to introduce this framework. + +## **3.8 Implementation Milestones** + +| Delivery milestone | What is delivered | +| :--- | :--- | +| 1. MVP: Basic plugin execution | A site admin creates, approves, and enables a digest-pinned, unprivileged container plugin for OnDemand validation. Scout runs it sequentially, provides the standard input, reads the standard result, and records timeout, logs, status, and result. The MVP includes basic registry and timeout validation, and supports plugin-owned adapters, but no special capabilities, input files, resource limits, full-host access, or new UI/CLI workflow. | +| 2. Hardware and policy controls | Add `inputFiles`, resource limits, approved capabilities such as `gpu-read` and `host-journal-read`, complete policy validation, audit details, and the site-admin workflow for an existing tool packaged with an adapter. | +| 3. Production rollout | Show plugin details in operator tools, add metrics, emergency disable, and recovery handling for full-host access, test in site environments, then enable approved plugins for discovery validation. | + +Parallel execution is not part of these delivery milestones. It can be added later when the framework supports it. + +# **4. Technical Considerations** + +## **4.1 Security** + +For every milestone, plugin execution is allowed only after all applicable checks pass: + +1. The image uses a digest, comes from a site-approved registry, and is acquired within the attempt deadline. +2. The site admin approves the complete revision and the API records matching verified state before it is enabled or launched. MVP approval covers the image digest, entrypoint, timeout, and unprivileged profile; later revisions also cover input files, capabilities, resource limits, and privileged or host-access settings. +3. A `hostAccess: full` revision has separate matching full-host approval before it is enabled or launched. +4. The timeout and any requested resource limits are permitted by site policy. + +The site admin is the normal author, approver, and enabler of a plugin revision. NICo validates policy and records the resulting server-managed state; it does not make the approval decision. Only a verified revision can be enabled. `hostAccess: full` always requires a separate full-host approval and can be limited to site-defined contexts or named machines. Emergency disable is allowed to the operator role and is audit logged. + +The framework places no raw secret values in plugin input. Plugin logs and results are untrusted and may contain secrets, so they are size-limited, redacted before persistence or display, and access-controlled. When redaction detects secret-like content, the system stores the redacted value, records a redaction event, and warns the operator; it never stores the detected raw value. Viewing diagnostic output requires the same or stronger permission as viewing the corresponding validation run. Every run records the exact image digest and plugin revision used. + +The MVP requires a digest-pinned image, basic registry and timeout validation, approved and enabled matching revision state, bounded image acquisition, and the standard unprivileged network-disabled runtime profile. Additional policy controls and capabilities are added in milestone 2. + +## **4.2 Observability** + +Logs, metrics, and audit events identify the run, plugin name, plugin revision, and image digest. Operators must be able to distinguish: + +1. A plugin that completed and found a machine problem (`fail`). +2. A plugin that could not complete the check (`error`). +3. A framework problem such as an image pull failure, timeout, or invalid result. + +## **4.3 Acceptance Criteria** + +1. An authorized site admin can configure and approve a digest-pinned container plugin for production use. +2. Machine Validation selects the plugin only for matching machines and contexts. +3. Scout provides the versioned standard input file and accepts only a size-bounded, valid result from a normally completed process. +4. An image-pull timeout, timeout, invalid result, or plugin crash produces a clear terminal state. +5. Operators can see the plugin revision, image digest, and outcome for a run. +6. Existing non-plugin Machine Validation tests continue to work. +7. Plugins run unprivileged by default, with networking disabled for every plugin profile. A verified revision can explicitly request `privileged: true`. `hostAccess: full` always requires separate approval, is further limited by site policy, and runs with the documented writable `/host` mount. +8. A stale or crashed plugin attempt is terminalized by reconciliation and does not leave the machine in validation. +9. Emergency disable rejects new launches and cancels active attempts for the affected revision. +10. A canceled full-host attempt marks the machine for operator recovery before it returns to service. +11. Before parallel execution is enabled, the scheduler enforces unique per-attempt working directories, concurrency limits, and exclusive-resource locking. + +# **5. End-to-End Site Admin Examples** + +## **5.1 Simple Platform Policy Plugin (MVP)** + +The site admin for `example-ai-west-prod` wants to allow only approved machine platforms during OnDemand validation. This plugin reads the standard input file, checks the platform against its configured parameters, and writes the standard result file. It does not need an adapter or special runtime access. + +1. Set the site policy. The site admin updates `nico-api-site-config.toml` in the site's deployment overlay, then applies the normal site-configuration rollout. This policy permits the site's registry and the required timeout. + + ```toml + [machine_validation_config.plugin_policy] + approved_registries = ["registry.example.com"] + max_timeout_seconds = 3600 + ``` + +2. Publish the plugin image. The site admin builds the image and publishes its immutable digest to the approved registry. + +3. Submit the catalog entry. The site admin saves the following manifest in the site's configuration repository and submits it through the Machine Validation API. The API confirms that the authenticated site is `example-ai-west-prod`, validates the request against `nico-api-site-config.toml`, and stores the disabled revision in the Machine Validation database. UI and CLI support are added later. + + ```yaml + site: example-ai-west-prod + name: platform-policy + runWhen: + contexts: [OnDemand] + image: registry.example.com/example-ai-west-prod/platform-policy@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef # Illustrative digest + entrypoint: ["/plugin/entrypoint"] + timeoutSeconds: 300 + order: 100 + parameters: + allowedPlatforms: [HGX-B200] + allowParallel: false + privileged: false + hostAccess: none + ``` + +4. Approve and enable it. The API validates the revision against policy. The site admin approves it, the API records verified state for that exact revision and digest, and the site admin enables it. The create request cannot set verification or enablement state. + +5. Run and record it. For a matching machine, Machine Validation snapshots the revision into a run item. Scout mounts the standard container paths, starts `/plugin/entrypoint`, and reads `/opt/nico/mv/output/result.json`. The API records the result alongside existing validation results. + +## **5.2 Existing GPU Health Tool with an Adapter (Milestone 2)** + +The site already has a GPU health tool that runs `./health-tool run --config operator.yaml`. It writes its own JSON to stdout and uses its own exit codes, so it cannot be used directly as a plugin. The site packages it with a small adapter; the framework remains unaware of the underlying tool. + +1. Set the site policy. The site admin updates `nico-api-site-config.toml` in the site's deployment overlay, then applies the normal site-configuration rollout. This policy permits the trusted registry plus read-only GPU and host-journal access. It does not allow privileged or full-host plugins. + + ```toml + [machine_validation_config.plugin_policy] + approved_registries = ["registry.example.com"] + allowed_capabilities = ["gpu-read", "host-journal-read"] + allow_privileged_plugins = false + allow_full_host_plugins = false + max_timeout_seconds = 3600 + ``` + +2. Build and publish the plugin adapter image. The image contains the health tool, its configuration reader, and `/plugin/entrypoint`. The adapter runs the tool, captures its tool-specific output, and writes the standard `/opt/nico/mv/output/result.json`. This is plugin-owned code, not Scout behavior. The site admin publishes its immutable digest to the approved registry. + +3. Submit the catalog entry. The site admin saves the manifest in the site's configuration repository and submits it through the Machine Validation API. The API confirms that the authenticated site is `example-ai-west-prod`, validates it against `nico-api-site-config.toml`, and stores the existing `operator.yaml` as an immutable input file with the disabled plugin revision. UI and CLI support are added later. + + ```yaml + site: example-ai-west-prod + name: gpu-health + runWhen: + contexts: [Discovery, OnDemand] + platforms: [HGX-B200] + image: registry.example.com/example-ai-west-prod/gpu-health-plugin@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef # Illustrative digest + entrypoint: ["/plugin/entrypoint"] + timeoutSeconds: 1800 + order: 200 + inputFiles: + operator.yaml: | + checks: + gpu_count: + expected_gpu_count: 8 + capabilities: [gpu-read, host-journal-read] + allowParallel: false + privileged: false + hostAccess: none + ``` + +4. Approve and enable it. The API validates the revision against policy. The site admin approves it, the API records verified state for that exact revision and digest, and the site admin enables it. The create request cannot set verification or enablement state. + +5. Run and translate the result. Scout writes `/opt/nico/mv/input/input.json` and `/opt/nico/mv/input/operator.yaml`, applies the approved capabilities, and starts `/plugin/entrypoint`. The adapter runs: + + ```text + ./health-tool run --config /opt/nico/mv/input/operator.yaml + ``` + + It maps the tool's success, unhealthy, and tool-error outcomes to `pass`, `fail`, and `error`, respectively. Missing or malformed tool output is always `error`, never `pass`. It preserves any advisory state in the optional `severity` field, then writes `/opt/nico/mv/output/result.json`. + +6. Operate it like every other plugin. Machine Validation validates and records the standard result. Operators see the plugin revision, image digest, outcome, severity, and diagnostic output without needing to know the underlying tool.