diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/.openspec.yaml b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/.openspec.yaml new file mode 100644 index 00000000000..548fa1a49fb --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/.openspec.yaml @@ -0,0 +1,2 @@ +schema: rhdh-spec-driven +created: 2026-08-25 diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/audit.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/audit.md new file mode 100644 index 00000000000..35a0fb579e0 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/audit.md @@ -0,0 +1,29 @@ +## Audit Report: mcp-registry-provider + +**Last audited:** 2026-08-27T18:25:00Z + +### Summary + +| Category | CRITICAL | WARNING | SUGGESTION | +| -------------------------- | -------- | ------- | ---------- | +| A — Entity propagation | 0 | 0 | 0 | +| B — Enum / vocabulary | 0 | 0 | 0 | +| C — Semantic contradiction | 0 | 0 | 0 | +| D — Codebase & convention | 0 | 0 | 0 | +| E — Namespace & ownership | 0 | 0 | 0 | +| F — Template / copy-paste | 0 | 0 | 0 | +| G — Extended coherence | 0 | 0 | 0 | +| H — Security lint | 0 | 0 | 0 | +| **Total** | **0** | **0** | **0** | + +### CRITICAL + +- None + +### WARNING + +- None + +### SUGGESTION + +- None diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/design.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/design.md new file mode 100644 index 00000000000..05d2c15a9a3 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/design.md @@ -0,0 +1,102 @@ +## Canonical Touchpoints + +Carried forward from the proposal: + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None + +No canonical document updates. This change introduces a new capability only and does not modify any existing canonical document or long-lived spec. It **consumes** the sibling [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/design.md) transform contract without altering it. + +## Context + +[`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) defines a pure, deterministic `server.json` → `mcp-server` `API` entity transform and explicitly scopes out ingestion. This change supplies the runtime that calls that transform: a Backstage catalog **entity provider** that fetches servers from an [MCP Registry](https://github.com/modelcontextprotocol/registry) on a schedule and populates the catalog. + +**Reference prototype.** The [`mcp-reg-proxy-proto`](https://github.com/gabemontero/rhdh-plugins/tree/mcp-reg-proxy-proto) branch of `rhdh-plugins` (plugin at `workspaces/ai-integrations/plugins/mcp-registry-proxy-backend/`) is a _proxy_ — it re-exposes registry endpoints through RHDH. It is a **design reference only**; this change is a _provider_ (one-way ingestion into the catalog), not a proxy. Two lessons carry over: + +1. **Config shape.** The prototype placed config under `mcp.registry.proxy.` reading `baseUrl` and `registryVersion`. This change instead uses the idiomatic Backstage catalog-provider location `catalog.providers.mcpRegistry.` (see D1). +2. **Pagination gap.** The prototype defined a `PaginatedResponse` type but **only fetched the first page** — it never followed `nextCursor`. This change treats full cursor traversal as a first-class requirement (see D4). + +**Registry API.** Per the [generic registry API](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/api/generic-registry-api.md#basic-example-list-servers), `GET //servers?cursor=&limit=` returns: + +```json +{ + "servers": [ { "server": { "name": "...", "description": "...", "version": "..." }, "_meta": { ... } } ], + "metadata": { "count": 10, "nextCursor": "..." } +} +``` + +Pagination is cursor-based: omit `cursor` on the first request; pass the prior `metadata.nextCursor` on each subsequent request; stop when it is absent/empty. Cursors are opaque. + +**Backstage integration.** The plugin is a `catalog-backend-module` that registers one `EntityProvider` per configured registry instance and schedules each via `SchedulerService`. On each tick it lists servers, maps them, and calls `connection.applyMutation({ type: 'full', entities })`. + +## Goals / Non-Goals + +**Goals:** + +- A scheduled catalog entity provider that ingests MCP servers from one or more registries into the RHDH catalog as `mcp-server` `API` entities. +- Complete ingestion via full cursor pagination (the gap left by the reference prototype). +- Idiomatic, upstream-aligned configuration (`catalog.providers.mcpRegistry.`) supporting multiple registries. +- Clean delegation to `mcp-registry-server-mapping` — the provider never reimplements the transform. +- Full-mutation semantics so the catalog converges to the registry's current state (adds, updates, prunes). +- Resilient, predictable (agent-native) error handling: skip a bad entry, fail a bad run without corrupting catalog state. + +**Non-Goals:** + +- The `server.json` → entity transform, annotation projection, secret redaction (owned by `mcp-registry-server-mapping`). +- A registry proxy / pass-through API. +- Registry authentication / write access, or per-registry credentials. +- Runtime invocation, health checking, or tool discovery of ingested servers. +- Cross-registry dedup/merge of the same server. +- Frontend / catalog UI changes. + +## Decisions + +### D1: Configuration under `catalog.providers.mcpRegistry.` (keyed map, multi-registry) + +Config lives at `catalog.providers.mcpRegistry.`, a keyed map where each key registers one provider instance with `baseUrl` (required), `apiVersion` (default `v1`), `schedule` (`SchedulerServiceTaskScheduleDefinition`), and `defaultOwner` (entity ref). This is the idiomatic Backstage entity-provider location (mirrors `catalog.providers.github.`, etc.), which upstream tooling and operators already understand. **Alternatives considered:** (a) the prototype's `mcp.registry.proxy.` namespace — rejected; that namespace reads as proxy/pass-through config and is not where catalog operators look for entity providers. (b) a flat single-registry block — rejected; a keyed map supports multiple registries at no extra cost and matches upstream providers. A `config.d.ts` declares the schema so app-config validation and IDE assistance work. + +### D2: Package as a `catalog-backend-module`, one `EntityProvider` per instance + +The plugin is a backend module registered via `createBackendModule` that extends the catalog via `catalogProcessingExtensionPoint.addEntityProvider(...)`, adding one `EntityProvider` per configured ``. Each provider owns its `getProviderName()` (e.g. `mcp-registry-provider:`), which becomes the entities' `locationKey` and drives full-mutation pruning scoping. **Alternatives considered:** (a) a single provider handling all registries internally — rejected; per-instance providers give independent schedules, independent failure isolation, and correct per-source `locationKey` pruning. (b) a standalone backend plugin with its own router — rejected; ingestion needs the catalog extension point, not an HTTP surface (that would be the proxy pattern). + +### D3: Schedule via `SchedulerService`; per-instance `schedule` with a documented default + +Each provider is driven by `scheduler.createScheduledTaskRunner(schedule)` and refreshes on the configured `SchedulerServiceTaskScheduleDefinition`. When `schedule` is omitted, a documented default (e.g. `frequency: { minutes: 30 }`, `timeout: { minutes: 3 }`) is applied rather than failing — a missing schedule should not block ingestion. Providers connect via the standard `EntityProvider.connect` + scheduled `run()` pattern. **Alternative:** require `schedule` and fail if absent — rejected; a sensible default keeps first-run setup simple, consistent with the mapping's "never fail for a supplyable default" stance. + +### D4: Full cursor pagination is mandatory + +Unlike the reference prototype (first page only), the provider loops: request `//servers`, accumulate `servers[]`, read `metadata.nextCursor`, and re-request with `?cursor=` until the cursor is absent/empty. Cursors are opaque and passed verbatim. A **loop safeguard** (max-pages / max-total bound, plus detecting a repeated cursor) prevents a misbehaving registry from spinning forever; hitting the bound fails the run (D6) rather than committing a partial catalog. **Alternative:** trust a single page (prototype behavior) — rejected; silently truncates ingestion for any registry larger than one page. + +### D5: Delegate wholly to `mcp-registry-server-mapping`; supply `defaultOwner` as the caller override + +For each server the provider calls the mapping transform, passing the instance's `defaultOwner` as the caller-override owner default (and any future caller defaults like `lifecycle`). The provider adds only provider-level concerns on top of the transform's output: the managed-by-location annotation / `locationKey` for catalog attribution and pruning. It never re-derives names, annotations, or `spec.remotes`. **Consequence:** the mapping is the single source of truth for entity shape; a mapping change automatically flows through the provider. **Alternative:** inline a copy of the mapping for "performance" — rejected; violates the single-contract goal and would drift. + +### D6: Failure isolation — skip bad entries, fail bad runs atomically + +Two failure tiers: (a) a single server that the mapping rejects (missing required `server.json` field) is logged with an identifying message and skipped; the run proceeds and commits the rest. (b) A registry-level error (unreachable, non-2xx, unparseable body, or pagination-safeguard trip) fails the whole run: **no** `applyMutation` is emitted, so the last-good catalog state is preserved, and the next scheduled tick retries. This matches the agent-native principle (predictable errors) and the full-mutation model (a partial full mutation would wrongly prune healthy entities). **Alternative:** commit whatever was fetched before an error — rejected; a partial full mutation prunes entities that still exist, causing catalog flapping. + +### D7: API-version slug is configurable, defaults to `v1`, discrepancy documented + +The endpoint is `//servers` with `apiVersion` defaulting to `v1`. The current reference registry actually serves `/v0` (prototype) or `/v0.1` (docs); the default follows the proposal's stated `v1` and operators override `apiVersion` to match their registry. URL joining normalizes trailing/leading slashes so `baseUrl` with or without a trailing `/` yields exactly one separator. This is captured as a risk below and an open question. **Alternative:** default `v0` to match today's registry — considered and deferred to the user's explicit choice of `v1`. + +## Risks / Trade-offs + +- **apiVersion default (`v1`) does not match the live registry (`/v0`, `/v0.1`)** → Operators must set `apiVersion` to their registry's actual version; the default is documented and overridable, and startup/first-sync errors name the endpoint that was requested. Revisit the default if the registry standardizes on a version. +- **Non-terminating or repeating cursor from a buggy registry** → D4 loop safeguard (max pages/total + repeated-cursor detection) trips and fails the run (D6) rather than looping forever or committing a partial catalog. +- **Partial-page fetch failure mid-pagination** → D6 fails the whole run with no mutation, preserving prior catalog state; no partial full mutation is ever committed. +- **`metadata.name` collisions across registries** (same name+version from two registries) → Out of scope (carried from the mapping's non-goals); per-instance `locationKey` keeps each provider's entities attributable, but true dedup/merge is deferred. Documented for a future change. +- **Large registries** → Pagination handles arbitrary size, but a very large registry produces a large full mutation each tick; `limit` tuning and schedule cadence are the operator's levers. Batching/streaming the mutation is a possible future optimization. +- **Mapping contract drift** → The provider depends on `mcp-registry-server-mapping`; because it delegates wholly (D5), a mapping change flows through automatically, but a breaking signature change to the transform would require a coordinated update here. +- **Unauthenticated registry assumption** → Auth is a non-goal; a registry requiring credentials will fail at fetch (D6) until a future auth extension lands. + +## Migration Plan + +Not applicable to existing data — this is additive and introduces no migration of prior state. Deployment: publish the backend module package, add it to the backend via `backend.add(...)`, and configure at least one `catalog.providers.mcpRegistry.` with a `baseUrl`. Rollback: remove the module registration (or the config block); ingested entities are pruned on the next catalog reconciliation because they are provider-managed via `locationKey`. The provider is inert when unconfigured, so shipping the package without config is a safe no-op. + +## Open Questions + +- **apiVersion default** — should the default track the live registry (`v0`/`v0.1`) instead of `v1` once the registry's versioning stabilizes? Currently `v1` per the proposal; revisit when the MCP Registry pins a stable API version. +- **`limit` / page-size configuration** — expose a per-instance `limit` (and pagination safeguard bounds) as config, or keep them internal constants? Deferred until real registry sizes inform sensible defaults. +- **Registry authentication** — token/header auth per instance is out of scope now; what shape (static token, `${ENV}` substitution, Backstage auth integration) should it take when added? +- **Cross-registry dedup/merge** — deferred (carried from the mapping change); needed if the same server is published to multiple configured registries. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/journal.jsonl b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/journal.jsonl new file mode 100644 index 00000000000..e96ed869545 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/journal.jsonl @@ -0,0 +1,13 @@ +{"ts":"2026-08-26T03:49:29Z","phase":"proposal","event":"change.created"} +{"ts":"2026-08-26T03:49:30Z","phase":"any","event":"turn.start","input":"Propose mcp-registry-provider: backend provider plugin consuming MCP Registry, mapping server.json to mcp-server API entities, with config baseUrl/schedule/apiVersion/defaultOwner + cursor pagination"} +{"ts":"2026-08-26T03:55:01Z","phase":"any","event":"decision","input":"config placement + apiVersion default","output":"catalog.providers.mcpRegistry. keyed multi-registry map; apiVersion defaults v1 with v0/v0.1 discrepancy documented as risk"} +{"ts":"2026-08-26T03:55:01Z","phase":"any","event":"decision","input":"reference proxy prototype gap","output":"proxy proto never implemented cursor pagination; provider MUST traverse metadata.nextCursor. Provider = catalog entity provider, not proxy"} +{"ts":"2026-08-26T03:58:49Z","phase":"authoring","event":"artifact.added","ref":"proposal.md"} +{"ts":"2026-08-26T03:58:49Z","phase":"authoring","event":"artifact.added","ref":"specs/mcp-registry-provider/spec.md"} +{"ts":"2026-08-26T04:07:07Z","phase":"authoring","event":"artifact.added","ref":"design.md"} +{"ts":"2026-08-26T04:07:07Z","phase":"authoring","event":"artifact.added","ref":"tasks.md"} +{"ts":"2026-08-26T04:07:13Z","phase":"any","event":"turn.end","output":"Created mcp-registry-provider change: proposal, spec, design, tasks. Config catalog.providers.mcpRegistry., cursor pagination, full mutation, delegates to mapping"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"turn.start","input":"/opsx:audit mcp-registry-provider"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"skill.invoked","name":"opsx:audit"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL): 1 WARNING (schedule type name drift), 1 SUGGESTION (cursor safeguard wording). User chose report-only. Wrote audit.md"} diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/proposal.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/proposal.md new file mode 100644 index 00000000000..c097961f6a9 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/proposal.md @@ -0,0 +1,51 @@ +## Why + +The [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) change defines the pure `server.json` → `mcp-server` `API` entity transform, but explicitly leaves ingestion out of scope: nothing yet fetches entries from an [MCP Registry](https://github.com/modelcontextprotocol/registry) and puts the resulting entities into the Backstage catalog. Without a provider, an operator who points RHDH at a registry gets no catalog entities. This change delivers that missing runtime component — a Backstage catalog **entity provider** that periodically reads a registry's `server.json` entries and populates the catalog with `mcp-server` `API` entities — so MCP servers published to a registry become discoverable in RHDH. + +## What Changes + +- Introduce a **backend catalog entity provider plugin** (a Backstage `catalog-backend-module`) that, on a configured schedule, lists MCP servers from a configured registry and applies the [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) transform to produce `mcp-server` `API` entities, then commits them to the catalog as a **full mutation** (so servers removed from the registry are pruned). +- Support **multiple registry instances** via idiomatic Backstage entity-provider configuration under `catalog.providers.mcpRegistry.`, each instance configuring: + - `baseUrl` — the URL of the MCP Registry (**required**). + - `schedule` — sync frequency as a standard `SchedulerServiceTaskScheduleDefinition` (`frequency`, `timeout`, optional `initialDelay`). + - `apiVersion` — the version segment used in the API endpoint slug; **defaults to `v1`**. + - `defaultOwner` — the default `spec.owner` (a `User`/`Group` entity reference) applied to every produced `API` entity, passed as the caller-override default into the mapping. +- Implement **cursor pagination**: the servers endpoint (`//servers`) is traversed by passing the prior response's `metadata.nextCursor` as the `cursor` query parameter until the cursor is absent/empty (per the [generic registry API](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/api/generic-registry-api.md#basic-example-list-servers)), so all servers are ingested regardless of page size. +- Specify **resilient, agent-native sync behavior**: a single server entry that fails to map is logged and skipped without aborting the run; a registry transport/protocol error fails that sync run (leaving the prior catalog state intact) and is retried on the next scheduled tick. + +## Capabilities + +### New Capabilities + +- `mcp-registry-provider`: A scheduled Backstage catalog entity provider that reads MCP servers from one or more configured MCP Registries (with cursor pagination), maps each `server.json` to an `mcp-server` `API` entity via [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md), and commits them to the catalog as a full mutation — including configuration (`catalog.providers.mcpRegistry.`), scheduling, API-version slug construction, and error handling. + +### Modified Capabilities + +_(none — no long-lived specs exist under `openspec/specs/` yet; this change introduces a new capability and **consumes** the sibling `mcp-registry-server-mapping` capability as its transform.)_ + +## Non-goals + +- **The mapping itself.** The `server.json` → entity transform, annotation projection, secret redaction, and identity/name rules are owned by [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) and consumed here unchanged. +- **A registry proxy or pass-through API.** Unlike the [reference proxy prototype](https://github.com/gabemontero/rhdh-plugins/tree/mcp-reg-proxy-proto), this plugin does not expose registry endpoints through RHDH; it is a one-way ingestion provider into the catalog. +- **Registry authentication / write access.** Assumes an unauthenticated (or externally-fronted) read-only registry endpoint; per-registry auth credentials are a future extension. +- **Runtime invocation, health checking, or tool discovery** of the ingested MCP servers. +- **Cross-registry deduplication / merge** of the same server published to multiple registries (carried over from the mapping's non-goals). +- **Frontend / catalog UI** changes; produced entities render via existing upstream `mcp-server` `API` entity support. + +## Canonical Touchpoints + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None (new capability only; `openspec/specs/` does not yet exist) + +**Change type**: feature-spec + +## Impact + +- **Depends on the sibling `mcp-registry-server-mapping` change** for the transform contract; this provider is the first consumer of that mapping and passes `defaultOwner` as the caller-override owner default. +- **Depends on Backstage backend framework**: the catalog `EntityProvider` interface, `SchedulerService` (`SchedulerServiceTaskScheduleDefinition`), the new backend system (`createBackendModule` / `coreServices`), and `RootConfigService` for reading `catalog.providers.mcpRegistry.`. +- **Source API**: MCP Registry generic API — `GET //servers?cursor=`; response `{ servers: [...], metadata: { count, nextCursor } }`. Cursors are opaque and traversed until absent. +- **API-version discrepancy** (documented risk): the current reference registry serves `/v0` (proxy prototype) / `/v0.1` (docs), while `apiVersion` defaults to `v1` per this proposal; operators override `apiVersion` to match their registry. +- **Consumers**: RHDH operators who configure a registry; developers and AI agents who then discover MCP servers via catalog search/filter over the `mcp-server` entities and their `modelcontextprotocol.io/*` annotations. +- **Packaging**: a new backend plugin package (Backstage catalog-backend-module naming convention), wired into the backend via `backend.add(...)`. +- **Upstream**: keep aligned with Backstage's entity-provider / scheduler APIs and the MCP Registry generic API as both evolve. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md new file mode 100644 index 00000000000..98b4efc7879 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md @@ -0,0 +1,129 @@ +## MCP Registry Provider + +This capability defines a Backstage catalog **entity provider** that ingests MCP servers from one or more configured [MCP Registries](https://github.com/modelcontextprotocol/registry) into the RHDH catalog as `mcp-server` `API` entities. + +On a configured schedule, each provider instance lists the registry's servers (`GET //servers`), traverses all pages via cursor pagination, transforms each `server.json` document into an `mcp-server` `API` entity using the [`mcp-registry-server-mapping`](../../../mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md) contract (supplying the configured `defaultOwner` as the caller-override owner), and commits the full set to the catalog as a single full mutation so that servers removed from the registry are pruned. + +This spec covers configuration, scheduling, registry API interaction (pagination and API-version slug construction), delegation to the mapping transform, catalog mutation semantics, and error handling. It does **not** redefine the `server.json` → entity transform, which is owned by `mcp-registry-server-mapping`. + +--- + +## ADDED Requirements + +### Requirement: Configure MCP registry provider instances + +The provider SHALL read its configuration from `catalog.providers.mcpRegistry`, treated as a keyed map where each key is a caller-chosen instance `` and each value configures one registry. For each instance the provider SHALL read `baseUrl` (**required**), `apiVersion` (optional, defaulting to the constant `v1`), `schedule` (optional, a standard `SchedulerServiceTaskScheduleDefinition`), and `defaultOwner` (optional, a Backstage entity reference). The provider SHALL register one independent provider instance per configured ``. When `catalog.providers.mcpRegistry` is absent, the provider SHALL register nothing and SHALL NOT error (the module is inert unless configured). + +#### Scenario: Single registry instance configured + +- **WHEN** `catalog.providers.mcpRegistry.redhatEcosystem` is configured with a `baseUrl` and a `schedule` +- **THEN** exactly one provider instance is registered for the `redhatEcosystem` id, using the configured `baseUrl` and `schedule`, `apiVersion` defaulting to `v1`, and no owner override beyond the mapping's own default + +#### Scenario: Multiple registry instances configured + +- **WHEN** `catalog.providers.mcpRegistry` contains two keys `internal` and `public`, each with its own `baseUrl` and `schedule` +- **THEN** two independent provider instances are registered, each syncing its own registry on its own schedule + +#### Scenario: Missing required baseUrl fails fast + +- **WHEN** a configured instance omits `baseUrl` +- **THEN** provider startup fails with an actionable error that names the offending instance `` and the missing `baseUrl` key + +#### Scenario: No configuration present + +- **WHEN** `catalog.providers.mcpRegistry` is not present in app-config +- **THEN** the module registers no provider instances and startup succeeds without error + +### Requirement: Sync on the configured schedule + +Each provider instance SHALL run its ingestion sync on the configured `schedule` using the Backstage `SchedulerService`. When `schedule` is omitted for an instance, the provider SHALL apply a documented default `SchedulerServiceTaskScheduleDefinition` rather than failing. The provider SHALL also perform an initial sync according to the schedule's `initialDelay` (or immediately when unset) after registration. + +#### Scenario: Scheduled sync runs at the configured frequency + +- **WHEN** an instance is configured with a `schedule` of `frequency: { minutes: 30 }` +- **THEN** the provider runs a full ingestion sync approximately every 30 minutes via the scheduler, independently of other instances + +#### Scenario: Schedule omitted uses the default + +- **WHEN** an instance is configured without a `schedule` +- **THEN** the provider applies the documented default schedule and syncs on that cadence without error + +### Requirement: List all registry servers with cursor pagination + +During a sync, the provider SHALL request the registry's servers from `//servers` and SHALL traverse every page using the registry's cursor pagination: it SHALL read `metadata.nextCursor` from each response and, when that value is present and non-empty, issue the next request with that value as the `cursor` query parameter, repeating until `metadata.nextCursor` is absent, null, or empty. Cursors SHALL be treated as opaque strings (never constructed or modified). All `servers[]` entries across all pages SHALL be accumulated for the sync. The provider SHALL guard against a non-terminating cursor loop with a documented safeguard. + +#### Scenario: Multi-page traversal + +- **WHEN** the registry returns a first page with `metadata.nextCursor` set and a second page with no `nextCursor` +- **THEN** the provider fetches both pages, passing the first page's `nextCursor` as the `cursor` parameter on the second request, and accumulates the servers from both pages + +#### Scenario: Single-page result + +- **WHEN** the registry returns a page whose `metadata.nextCursor` is absent or empty +- **THEN** the provider stops after that single request and processes only the accumulated servers + +#### Scenario: Opaque cursor is passed unchanged + +- **WHEN** a response's `metadata.nextCursor` is an opaque token +- **THEN** the provider passes that token verbatim as the `cursor` query parameter without parsing or altering it + +### Requirement: Construct the servers endpoint from apiVersion + +The provider SHALL construct the servers endpoint as `//servers`, where `apiVersion` is the configured value or the `v1` default. The provider SHALL join `baseUrl` and the version segment without duplicating or dropping path separators, regardless of whether `baseUrl` has a trailing slash. + +#### Scenario: Default apiVersion + +- **WHEN** an instance configures `baseUrl: https://registry.example.com` and no `apiVersion` +- **THEN** the provider requests `https://registry.example.com/v1/servers` + +#### Scenario: Overridden apiVersion + +- **WHEN** an instance configures `baseUrl: https://registry.example.com/` (trailing slash) and `apiVersion: v0` +- **THEN** the provider requests `https://registry.example.com/v0/servers` with exactly one separator between segments + +### Requirement: Map each registry server to an mcp-server API entity + +For every accumulated server entry, the provider SHALL extract its `server.json` document and produce an `mcp-server` `API` entity by applying the `mcp-registry-server-mapping` transform, supplying the instance's configured `defaultOwner` as the caller-override owner default. The provider SHALL NOT reimplement or alter the field mapping, annotation projection, or identity rules defined by `mcp-registry-server-mapping`. Each produced entity SHALL carry the provider's location/ownership annotations so the catalog attributes the entity to this provider instance. + +#### Scenario: Server mapped with configured default owner + +- **WHEN** a sync retrieves a `server.json` and the instance configures `defaultOwner: group:default/mcp-admins` +- **THEN** the produced `mcp-server` `API` entity has `spec.owner: group:default/mcp-admins` (the caller override), with all other fields set by the `mcp-registry-server-mapping` transform + +#### Scenario: Default owner omitted falls back to the mapping default + +- **WHEN** a sync retrieves a `server.json` and the instance configures no `defaultOwner` +- **THEN** the produced entity's `spec.owner` is the `mcp-registry-server-mapping` default (`unknown`) + +#### Scenario: Provider attribution annotations present + +- **WHEN** the provider produces an entity +- **THEN** the entity carries the provider instance's managed-location annotation so the catalog associates the entity with this provider and can prune it on removal + +### Requirement: Commit ingested entities as a full mutation + +At the end of each successful sync, the provider SHALL commit the complete set of produced entities to the catalog as a single **full** mutation (not incremental), so that entities for servers no longer present in the registry are removed from the catalog and re-added/updated entities reflect the latest `server.json`. The provider SHALL NOT emit a mutation for a sync run that failed to complete (see error handling), leaving the prior catalog state intact. + +#### Scenario: Removed server is pruned + +- **WHEN** a server present in a prior sync is absent from the current sync's accumulated servers +- **THEN** the current sync's full mutation omits that server's entity, and the catalog removes it + +#### Scenario: Updated server reflects latest state + +- **WHEN** a server's `server.json` changes between syncs (e.g. a new description) +- **THEN** the produced entity in the current full mutation reflects the latest `server.json` + +### Requirement: Resilient, agent-native error handling + +A single server entry that cannot be mapped (e.g. it omits a `server.json`-required field and the mapping rejects it) SHALL be logged with an actionable message identifying the entry and SHALL be skipped, without aborting the sync or discarding the other entries. A registry transport or protocol error (unreachable host, non-success HTTP status, or unparseable response body) SHALL fail the current sync run: the provider SHALL NOT commit a partial full mutation, SHALL log the error, and SHALL retry on the next scheduled tick, leaving the prior catalog state intact. + +#### Scenario: One malformed server does not abort the sync + +- **WHEN** one accumulated server entry fails mapping while the others succeed +- **THEN** the provider logs the failing entry, skips it, and commits a full mutation containing the successfully mapped entities + +#### Scenario: Registry fetch error aborts the run without a mutation + +- **WHEN** a page request returns a non-success HTTP status or the response body cannot be parsed +- **THEN** the provider logs the error, does not commit any mutation for this run, leaves the previously committed catalog entities intact, and retries on the next scheduled tick diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/tasks.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/tasks.md new file mode 100644 index 00000000000..23bb0d4b4ea --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/tasks.md @@ -0,0 +1,49 @@ + + + +## 1. Plugin Scaffolding & Packaging + +- [ ] 1.1 Create the backend `catalog-backend-module` plugin package (Backstage catalog-backend-module naming convention) with `package.json`, `tsconfig`, and lint config matching the workspace's plugin conventions +- [ ] 1.2 Add the `createBackendModule` skeleton that registers against `catalogProcessingExtensionPoint`, depending on `coreServices` (`rootConfig`, `logger`, `scheduler`) +- [ ] 1.3 Document installation in the plugin `README.md` (add via `backend.add(...)`, minimal app-config example) + +## 2. Configuration + +- [ ] 2.1 Author `config.d.ts` declaring `catalog.providers.mcpRegistry.` with `baseUrl` (required), `apiVersion?` (default `v1`), `schedule?` (`SchedulerServiceTaskScheduleDefinitionConfig`), and `defaultOwner?` +- [ ] 2.2 Implement config reading: parse `catalog.providers.mcpRegistry` as a keyed map into a typed per-instance config array; register nothing (no error) when the key is absent +- [ ] 2.3 Implement validation with actionable errors that name the offending instance `` (fail fast when `baseUrl` is missing); apply the `apiVersion` default (`v1`) and the documented default schedule when omitted +- [ ] 2.4 Add unit tests for config parsing/validation: single instance, multiple instances, missing `baseUrl`, and absent-config no-op + +## 3. Registry Client & Pagination + +- [ ] 3.1 Define the registry API response types (`servers[]`, `metadata.count`, `metadata.nextCursor`) and the `server.json` extraction from each `servers[]` entry (`.server`) +- [ ] 3.2 Implement servers-endpoint URL construction `//servers` with slash normalization (works with and without a trailing slash on `baseUrl`) +- [ ] 3.3 Implement cursor pagination: loop passing prior `metadata.nextCursor` as the `cursor` query param until it is absent/empty, accumulating all `servers[]`; treat cursors as opaque +- [ ] 3.4 Implement the pagination loop safeguard (max-pages/total bound + repeated-cursor detection) that fails the run rather than looping forever +- [ ] 3.5 Implement registry-error handling (unreachable host, non-2xx status, unparseable body) raising a typed error that aborts the run +- [ ] 3.6 Add unit tests for the client using mocked HTTP: single page, multi-page traversal, empty/absent cursor termination, opaque-cursor passthrough, and error/ safeguard cases + +## 4. Entity Provider & Scheduling + +- [ ] 4.1 Implement the `EntityProvider` class: `getProviderName()` = `mcp-registry-provider:`, `connect()` storing the connection, and a `run()` performing one sync +- [ ] 4.2 Wire scheduling via `SchedulerService.createScheduledTaskRunner(schedule)` per instance, honoring `initialDelay`; register one provider per configured `` +- [ ] 4.3 Implement the full-mutation commit: on successful sync call `connection.applyMutation({ type: 'full', entities })`; on a failed run emit no mutation (preserve prior catalog state) +- [ ] 4.4 Attach provider attribution to each entity (managed-by-location annotation / `locationKey`) so entities are scoped to this provider instance for pruning +- [ ] 4.5 Add unit tests: full mutation contents, pruning of removed servers across two syncs, updated server reflected, and no-mutation-on-failed-run + +## 5. Mapping Integration + +- [ ] 5.1 Depend on the sibling `mcp-registry-server-mapping` transform and invoke it per accumulated server, passing the instance's `defaultOwner` as the caller-override owner default (never reimplement the mapping) +- [ ] 5.2 Implement per-entry failure isolation: catch a mapping rejection (e.g. missing required `server.json` field), log an actionable message identifying the entry, skip it, and continue the run +- [ ] 5.3 Add integration tests over sample `server.json` inputs → produced `mcp-server` `API` entities, asserting `spec.owner` reflects `defaultOwner` (and the mapping default `unknown` when omitted), and that one bad entry does not abort the batch + +## 6. End-to-End Verification & Docs + +- [ ] 6.1 Add an end-to-end test wiring config → mocked paginated registry → mapping → full mutation, asserting the mutation converges to the registry's current server set +- [ ] 6.2 Verify produced entities pass the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`) — reusing the mapping change's conformance expectations +- [ ] 6.3 Verify the apiVersion discrepancy handling: default `v1` requests `/v1/servers` and an override (`v0`) is honored, with a documented note for operators +- [ ] 6.4 Finalize `README.md` / config docs: full `catalog.providers.mcpRegistry.` example (baseUrl, apiVersion, schedule, defaultOwner), pagination behavior, and error-handling semantics +- [ ] 6.5 Run the workspace lint, typecheck, and test suite; ensure the new package builds and passes CI conventions diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/.openspec.yaml b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/.openspec.yaml new file mode 100644 index 00000000000..ed871a87397 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/.openspec.yaml @@ -0,0 +1,2 @@ +schema: rhdh-spec-driven +created: 2026-08-18 diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/audit.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/audit.md new file mode 100644 index 00000000000..8cbc0601c2e --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/audit.md @@ -0,0 +1,45 @@ +## Audit Report: mcp-registry-server-mapping + +**Last audited:** 2026-08-25T00:03:43Z + +_Re-audit following the `repository.url`/`websiteUrl` link-mapping revision (repository.url now dual-emits `backstage.io/source-location` + a `metadata.links` "Source Code" entry; `websiteUrl` link titled "Website"). Fix-and-reaudit loop ran 2 passes; all findings from both passes were fixed and confirmed. No CRITICAL findings at any point in this run._ + +### Summary + +| Category | CRITICAL | WARNING | SUGGESTION | +| -------------------------------------- | -------- | ------- | ---------- | +| A (Entity propagation) | 0 | 0 | 0 | +| B (Enum / vocabulary) | 0 | 0 | 0 | +| C (Semantic contradiction) | 0 | 0 | 0 | +| D (Codebase & convention grounding) | 0 | 0 | 0 | +| E (Namespace & cross-change ownership) | 0 | 0 | 0 | +| F (Template / copy-paste residue) | 0 | 0 | 0 | +| G (Extended coherence) | 0 | 0 | 0 | +| H (Security lint) | 0 | 0 | 0 | +| **Total** | **0** | **0** | **0** | + +### CRITICAL + +- None + +### WARNING + +- None + +### SUGGESTION + +- None + +--- + +**Pass 1 findings (all resolved):** + +- **[A] WARNING**: `specs/mcp-registry-annotation-projection/spec.md:22` "not re-projected" scenario listed only `remotes[].url` as natively-mapped, while the server-mapping spec (line 29) and task 3.4 map/exclude both `type` and `url` → **Fixed**: now reads `remotes[].type`/`url`. +- **[C] WARNING**: Ambiguity over whether `repository.subfolder` is separately projected as `modelcontextprotocol.io/repository.subfolder` in addition to being combined into the source-location/link values → **Resolved (no change needed)**: subfolder is not in task 3.4's skip list, so it projects normally under the round-trip fidelity rule (annotation-projection spec); the combine-when-present behavior is already stated in the server-mapping spec, design D10, and tasks. Not re-reported in pass 2. + +**Pass 2 findings (all resolved):** + +- **[B] SUGGESTION**: `proposal.md:11` said `spec.lifecycle` defaults to `production` while owner used "constant `unknown`" and design/tasks/spec used "constant `production`" → **Fixed**: now "the constant `production`". +- **[A] SUGGESTION**: `proposal.md:8` source-location clause omitted the "(combined with `repository.subfolder` when present)" detail present in design/tasks/spec → **Fixed**: detail added. + +**Cross-change ownership (Category E):** `modelcontextprotocol.io/*` and the `mcp-server` API entity mapping are owned solely by this change. `backstage.io/source-location` is a standard Backstage annotation also used by sibling `aicontext-catalog-entity-kind` for its own `AIResource`/git entities (via `UrlReaderProcessor`); this change emits it for `mcp-server` entities. Different entity kinds, convergent standard usage — no exclusive-ownership conflict. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/design.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/design.md new file mode 100644 index 00000000000..ff682b23057 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/design.md @@ -0,0 +1,99 @@ +## Canonical Touchpoints + +Carried forward from the proposal: + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None + +No canonical document updates. This change introduces new capabilities only and does not modify any existing canonical document or long-lived spec. + +## Context + +MCP registries publish server entries as `server.json` documents conforming to the draft [`server.schema.json`](https://raw.githubusercontent.com/modelcontextprotocol/registry/refs/heads/main/docs/reference/server-json/draft/server.schema.json). Backstage already supports cataloging MCP servers as `API` entities with `spec.type: mcp-server` (implemented upstream in `backstage/backstage`, not by RHDH). This change defines the pure transform between the two — one `server.json` in, one `API` entity out — and nothing else. Who fetches registry entries and when (an entity provider, a polling schedule) is explicitly out of scope. + +Upstream Backstage defines a **dedicated `mcp-server` `API` entity schema** that overrides the base `API` schema for `spec.type: mcp-server`: + +- The **dedicated mcp-server entity schema** [`McpServerApiEntity.ts`](https://github.com/backstage/backstage/blob/f91434377dc43cd64bef82344e3f2b539bfdaf11/packages/catalog-model/src/kinds/McpServerApiEntity.ts#L28-L36) (Backstage [PR #34016](https://github.com/backstage/backstage/pull/34016)) does `Omit` and redefines `spec` for `spec.type: mcp-server` as required `type`, `lifecycle`, `owner`, and `remotes[]` (with optional `system`). `spec.remotes` is **required** and `spec.definition` is **not part of this schema** — the dedicated schema replaces (not merely extends) the base `spec`. +- The **canonical example** [`backstage-mcp-server-api.yaml`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml) matches that schema: top-level `spec.remotes[]` (`type`, `url`) and no `spec.definition`. +- The committed **base API schema** [`API.v1alpha1.schema.json`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json) lists `definition` among the required `spec` fields and does not define `spec.remotes` — but that `definition` requirement does **not** apply to `spec.type: mcp-server`, which is governed by the dedicated schema above. + +The mapping therefore targets the dedicated `mcp-server` entity shape: top-level `spec.remotes[]` in place of `spec.definition`. This aligns with the upstream-first principle — the shape is the one upstream ships, not an RHDH invention. + +## Goals / Non-Goals + +**Goals:** + +- A deterministic, idempotent, side-effect-free transform: `server.json` (+ caller defaults) → one `mcp-server` `API` entity. +- Faithful adherence to the upstream mcp-server entity shape (top-level `spec.remotes[]`, no `spec.definition`). +- Collision-free entity identity across multiple versions of the same server. +- Lossless capture of source data: every non-null scalar leaf is recoverable from a native field or a `modelcontextprotocol.io/*` annotation. +- Catalog-valid output: every produced key/name passes Backstage validation. + +**Non-Goals:** + +- Registry HTTP client, polling, scheduling, or an entity provider/processor (separate future change). +- Modifying the upstream `mcp-server` entity contract or its validation. +- Reverse mapping (entity → `server.json`) beyond the scalar round-trip guarantee. +- Executing or health-checking mapped servers, or interpreting local `packages[]` runtime details. +- Deduplicating the same server across multiple registries. + +## Decisions + +### D1: Target the upstream example shape — top-level `spec.remotes[]`, no `spec.definition` + +The entity carries `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, and top-level `spec.remotes[]` (`type`, `url`). No `spec.definition` is emitted. **Alternatives considered:** (a) inline `spec.definition` MCP Server Specification string — rejected; the mcp-server entity uses `spec.remotes` instead. (b) Dual-write both `spec.definition` and `spec.remotes` for base-schema safety — rejected as redundant given the target shape uses `spec.remotes`. + +### D2: Two-tier mapping — native lift, then annotation projection fallback + +Attributes with a native home lift into `metadata`/`spec` fields (`mcp-registry-server-mapping`); everything else is projected into `modelcontextprotocol.io/*` annotations (`mcp-registry-annotation-projection`). This keeps the entity idiomatic for catalog consumers while losing no source data. **Alternative:** stuff the entire `server.json` into a single annotation blob — rejected; opaque and not individually searchable/filterable. + +### D3: Annotation key encoding — dot-separated path after the prefix + +Nested paths are encoded as `modelcontextprotocol.io/attribute.tree.to.leaf` (object keys by name, array elements by zero-based index). Backstage annotation keys allow exactly one `/` and a ≤63-char name segment over a restricted character set, so path segments are sanitized (illegal characters and leading `_` replaced) and over-length keys are truncated with a stable hash suffix; sanitization collisions are disambiguated by the same hash suffix. **Alternatives considered:** literal slashes (`.../attr/tree/leaf`) — rejected, invalid Backstage keys; hyphenated scalars + JSON blobs for arrays — rejected, less uniform and less queryable. + +### D4: Entity identity — `metadata.name` = `__` + +A registry publishes one `server.json` per version and each becomes its own entity, so a name derived from the canonical name alone would collide across versions. `metadata.name` is the sanitized canonical name and sanitized version joined by `__`. The bare canonical name is preserved in `modelcontextprotocol.io/name` and the version in `modelcontextprotocol.io/version`, so both remain individually queryable and the identity is reconstructable. Over-length/collision falls back to truncation + stable hash suffix. **Alternative:** encode the version in `metadata.namespace` — rejected; fragments entity references and complicates relationships. + +### D5: Supplying fields absent from `server.json` — owner and lifecycle + +`server.json` (per the base `server.schema.json`) has no owner or lifecycle fields. `spec.owner` is set to the constant `unknown` by default; a caller MAY supply an override default, but the transform never fails for a missing owner (a placeholder owner keeps the output valid, and the future ingestion change can reassign ownership). `spec.lifecycle` is set to the constant `production` by default; a caller MAY supply an override default lifecycle value. Both fields use the same caller-override pattern for consistency. **Alternatives considered:** (a) require caller-provided owner/lifecycle and fail if absent — rejected; a pure transform should always yield a valid entity, and ownership/lifecycle assignment belongs to the ingestion layer. (b) derive lifecycle from a `status` field — rejected; `status` is not part of the base `server.schema.json` (verified 2026-08-21 against the draft schema). + +### D6: Determinism and idempotency + +The transform is a pure function of (`server.json`, caller defaults) with stable ordering of `spec.remotes`, `metadata.tags`, and annotation keys, and no timestamps or randomness. This makes the output safe as the identity for repeated ingestion and usable as a golden-file test oracle. + +### D7: Fail-open to generic projection on schema drift + +The draft `server.schema.json` evolves. Native mappings are pinned to known fields; any field not recognized by a native rule is still captured by the generic annotation projection. New/unknown source fields are therefore never dropped — at worst they land in annotations rather than a native field. + +### D8: Servers with no remotes emit an empty `spec.remotes: []` + +The upstream `McpServerApiEntity` schema requires `spec.remotes`. A `server.json` that declares no `remotes` (e.g. a local-`packages`-only server) is therefore mapped to an entity with an **empty array** `spec.remotes: []`, never an omitted field. This keeps the output both schema-conformant and deterministic (D6) — the no-remotes case has a single, stable representation. **Alternative:** treat a no-remotes server as a mapping failure — rejected; such servers are valid registry entries and their `packages`/metadata are still worth cataloging (their runtime details are preserved via annotation projection). **Dependency note:** this assumes `McpServerApiEntity` accepts an empty `remotes` array (no `minItems: 1`); if upstream later enforces a non-empty `remotes`, revisit this decision (emit a failure or a documented placeholder). + +### D9: Redact secret-flagged input values from annotation projection + +The `server.json` `Input` shape (used by `packages[].environmentVariables[]`, `remotes[].headers[]`, `remotes[].variables`, and package/runtime arguments) carries an `isSecret` flag alongside `default`/`value`. Because annotation projection (D2/D3) emits scalar leaves into **searchable, plaintext** catalog annotations, projecting the `default`/`value` of an `isSecret: true` input would publish a credential. **Decision:** when an input object declares `isSecret: true`, the projection walker SHALL prune (omit) that object's `default` and `value` leaves; all non-secret sibling leaves (`name`, `description`, `format`, `isRequired`, `isSecret` itself, `choices`, …) continue to project. The redaction applies uniformly to every `isSecret`-bearing input, not only environment variables — redacting env vars while leaving remote `headers`/`variables` exposed would reintroduce the same leak. **Round-trip consequence:** this is a deliberate exception to the scalar round-trip fidelity guarantee (D-note below and the projection spec); a pruned secret leaf is intentionally _not_ recoverable from the entity. **Alternatives considered:** (a) hash/mask the value instead of omitting — rejected; a mask still advertises the secret's presence and length without adding catalog value, and a hash is neither reversible nor useful for discovery. (b) project into a differently-prefixed "sensitive" annotation — rejected; catalog annotations are not a secret store, so any in-entity representation is unsafe. + +### D10: Repository emits both `backstage.io/source-location` and a titled `metadata.links` entry; `websiteUrl` link titled "Website" + +`repository.url` (combined with `repository.subfolder` when present) is emitted **both** as the canonical Backstage `backstage.io/source-location` annotation — the annotation source-aware Backstage tooling (source view, scaffolder, TechDocs) reads to locate an entity's repository — **and** as a human-visible `metadata.links` entry titled `Source Code`. `websiteUrl` is emitted as a `metadata.links` entry titled `Website`. Emitting the source-location annotation in addition to the link keeps the entity idiomatic for both machine consumers (the annotation) and the catalog UI (the titled link). **Alternatives considered:** (a) emit only the `metadata.links` source entry and omit `backstage.io/source-location` — rejected; without the canonical annotation, upstream source-location tooling cannot resolve the repository. (b) emit only the annotation and no link — rejected; the annotation is not surfaced as a browsable link in the catalog UI. The `backstage.io/source-location` value is set by the direct mapping and is therefore a reserved annotation that the generic projection (D2/D3) must not overwrite or re-derive. + +## Risks / Trade-offs + +- **63-char truncation collisions** → Deterministic hash suffix on truncation and on sanitization collisions keeps keys unique; the hash is derived from the full source path so it is stable across runs. +- **`metadata.name` collisions across registries** (same name+version from two registries) → Out of scope here (no dedup); documented so the future ingestion change can namespace or dedup. Within a single source the `__` + hash-suffix rule guarantees uniqueness. +- **Draft schema drift** → D7 fail-open projection; the mapping table is versioned against the draft and revisited when the schema changes. +- **Lossy flattening of deep `packages[]` config** → Accepted; runtime package details are preserved as scalar-leaf annotations for discoverability, not interpreted. Round-trip fidelity is guaranteed only for scalar leaves. +- **Secret leakage into searchable annotations** (remote `headers`/`variables`, `environmentVariables` carrying `default`/`value`) → D9 prunes the `default`/`value` leaves of any `isSecret: true` input from projection. This is a deliberate carve-out from scalar round-trip fidelity — secret leaves are intentionally unrecoverable from the entity. Non-secret metadata on the same input still projects, so discoverability is preserved. +- **Upstream shape may change** (RFC #32062 could reintroduce `spec.definition` or formalize `spec.remotes` in the base schema) → The target shape is isolated to `mcp-registry-server-mapping`; a shape change is a localized spec/mapping update. + +## Migration Plan + +Not applicable — new capabilities with no existing data or behavior to migrate. The mapping is additive and has no runtime deployment surface of its own until a future ingestion change consumes it. + +## Open Questions + +- Should the mapping optionally map the reverse-DNS namespace (portion before `/` in `server.json` `name`) to `metadata.namespace`, or keep a single default namespace? Deferred to the ingestion change, where entity-ref implications are clearer. +- Cross-registry dedup/merge of the same server (same name+version from multiple registries) — deferred to the ingestion change. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/journal.jsonl b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/journal.jsonl new file mode 100644 index 00000000000..720c34564a1 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/journal.jsonl @@ -0,0 +1,21 @@ +{"ts":"2026-08-18T14:25:44Z","phase":"proposal","event":"change.created","ref":"mcp-registry-server-mapping","output":"Scaffolded change for mapping MCP registry server.json entries to catalog API mcp-server entities"} +{"ts":"2026-08-18T14:25:44Z","phase":"any","event":"turn.start","input":"Create data mapping: MCP registry server.json draft schema to catalog API mcp-server entities; unmapped attrs to modelcontextprotocol.io annotations"} +{"ts":"2026-08-18T14:55:24Z","phase":"authoring","event":"artifact.added","ref":"proposal.md","output":"Proposal: server.json to mcp-server API entity mapping; target uses top-level spec.remotes (no spec.definition); unmapped attrs to modelcontextprotocol.io annotations"} +{"ts":"2026-08-18T15:57:35Z","phase":"authoring","event":"artifact.added","ref":"specs/mcp-registry-server-mapping/spec.md","output":"Direct mapping spec: server.json to mcp-server API entity; metadata.name=__; remotes to spec.remotes; owner/lifecycle supply; deterministic"} +{"ts":"2026-08-18T15:57:35Z","phase":"authoring","event":"artifact.added","ref":"specs/mcp-registry-annotation-projection/spec.md","output":"Annotation projection spec: dot-separated modelcontextprotocol.io keys, sanitization/63-char truncation, no-overwrite, scalar round-trip"} +{"ts":"2026-08-18T15:57:46Z","phase":"any","event":"decision","input":"Resolve mcp-server entity target shape and version-uniqueness for the mapping","output":"Target top-level spec.remotes (no spec.definition); dot-path modelcontextprotocol.io annotations; metadata.name=__; version to dedicated annotation"} +{"ts":"2026-08-18T16:04:16Z","phase":"authoring","event":"artifact.added","ref":"design.md","output":"Design: target upstream spec.remotes shape; 7 decisions incl dot-path annotations, name__version identity, owner/lifecycle supply, fail-open projection"} +{"ts":"2026-08-18T16:04:16Z","phase":"authoring","event":"artifact.added","ref":"tasks.md","output":"Tasks: mapping reference, direct field mapping, annotation projection, conformance fixtures/verification/docs"} +{"ts":"2026-08-18T16:04:16Z","phase":"any","event":"turn.end","output":"Created change mcp-registry-server-mapping with proposal, 2 specs, design, tasks; target spec.remotes (no definition), name__version identity, dot-path annotations"} +{"ts":"2026-08-21T22:33:53Z","phase":"any","event":"turn.start","input":"Remove status field mapping - not in base server.schema.json spec"} +{"ts":"2026-08-21T22:58:20Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/, design.md, tasks.md","input":"Remove status field - not in base server.schema.json per user finding","output":"Removed status→lifecycle derivation; lifecycle now defaults to production (caller-overridable)"} +{"ts":"2026-08-21T22:58:32Z","phase":"any","event":"turn.end","output":"Updated all artifacts to remove status field mapping (not in base schema); spec.lifecycle now defaults to production"} +{"ts":"2026-08-21T23:06:30Z","phase":"any","event":"turn.start","input":"Run pre-implementation audit on updated artifacts"} +{"ts":"2026-08-21T23:20:59Z","phase":"any","event":"turn.end","output":"Audit complete: all artifacts coherent, ready for implementation"} +{"ts":"2026-08-21T23:21:13Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-08-24T15:33:00Z","phase":"any","event":"turn.start","input":"Revise repository.url/websiteUrl link mapping: repository.url→both backstage.io/source-location and metadata.links; link titles Website/Source Code"} +{"ts":"2026-08-24T15:33:00Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/, design.md, tasks.md","input":"repository.url maps to both backstage.io/source-location and a metadata.links entry; websiteUrl link titled Website; repository.url link titled Source Code","output":"Reversed prior 'no source-location' rule; added D10; retitled homepage/source links to Website/Source Code across all artifacts"} +{"ts":"2026-08-24T15:33:00Z","phase":"any","event":"turn.end","output":"Updated all artifacts: repository.url now dual-emits source-location + metadata.links (Source Code); websiteUrl link titled Website"} +{"ts":"2026-08-25T00:03:43Z","phase":"any","event":"turn.start","input":"Re-run pre-implementation audit after repository.url/websiteUrl link-mapping revision"} +{"ts":"2026-08-25T00:03:43Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-08-25T00:03:43Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL). 2 passes; pass1 1 WARNING fixed (remotes type/url in projection scenario), pass2 2 SUGGESTIONs fixed (proposal wording alignment). Artifacts coherent."} diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/proposal.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/proposal.md new file mode 100644 index 00000000000..cdb5afe8c1c --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/proposal.md @@ -0,0 +1,75 @@ +## Why + +MCP registries (the upstream [MCP Registry](https://github.com/modelcontextprotocol/registry) and downstream/private mirrors) publish server entries as `server.json` documents conforming to the [`server.schema.json` draft](https://raw.githubusercontent.com/modelcontextprotocol/registry/refs/heads/main/docs/reference/server-json/draft/server.schema.json). Backstage already supports cataloging MCP servers as `API` entities with `spec.type: mcp-server` — defined upstream in [`backstage/backstage`](https://github.com/backstage/backstage) (Backstage RFC [#32062](https://github.com/backstage/backstage/issues/32062), not by RHDH) — but there is no defined, deterministic way to turn a registry `server.json` into such an entity. Without a canonical mapping, every ingestion path would invent its own field translation, lose registry metadata that has no native catalog home, and produce entities that do not round-trip. A single documented mapping contract makes registry ingestion predictable and lossless, and is the prerequisite for any future registry entity provider. + +## What Changes + +- Define a deterministic, idempotent **mapping contract** that transforms one `server.json` document (draft `server.schema.json`) into one Backstage `API` entity with `spec.type: mcp-server`, conformant to the upstream Backstage `mcp-server` `API` entity specification (defined in [`backstage/backstage`](https://github.com/backstage/backstage)). +- Specify the **direct field mapping**: which `server.json` attributes lift into native Backstage/entity fields — `metadata.name` (derived as `__` so each server version gets a collision-free entity), `metadata.title`/`description`, `metadata.links` (a `websiteUrl` entry titled `Website` and a `repository.url` entry titled `Source Code`, with `repository.url` (combined with `repository.subfolder` when present) also emitted as a `backstage.io/source-location` annotation), `spec.type`/`lifecycle`/`owner`, the top-level `spec.remotes[]` (`type`, `url`) that the upstream mcp-server entity uses in place of `spec.definition`, and dedicated `modelcontextprotocol.io/name` and `modelcontextprotocol.io/version` fields preserving the bare canonical name and version. +- Specify the **annotation projection** fallback: every `server.json` attribute that has no native home is projected into an annotation keyed `modelcontextprotocol.io/`, encoding nested objects and array indices as dot-separated segments within a single-slash, catalog-valid key. **Secret-flagged inputs** (`isSecret: true`) have their `default`/`value` leaves pruned so no credential is published into a searchable catalog annotation. +- Specify **catalog-valid key construction**: character sanitization, the 63-character name-segment limit with deterministic hash-suffix truncation, and collision disambiguation — because Backstage annotation keys permit only one `/` and a restricted name-segment character set. +- Specify **field-supply rules** for catalog-required fields absent from `server.json` (`spec.owner` defaults to the constant `unknown`, `spec.lifecycle` defaults to the constant `production` — both overridable by caller defaults, never a failure), and **round-trip fidelity** so scalar leaves are recoverable from the produced entity (except redacted secrets). +- Deliver a canonical mapping-table reference, worked examples, and a conformance fixture set (`server.json` input → expected entity output). + +## Capabilities + +### New Capabilities + +- `mcp-registry-server-mapping`: The deterministic direct field mapping from a `server.json` (draft `server.schema.json`) document to the native fields of an `mcp-server` `API` entity — identity/name sanitization with canonical-name preservation, descriptive metadata, `remotes` → top-level `spec.remotes[]`, and supply of catalog-required fields (`spec.owner`, `spec.lifecycle`) absent from the source. +- `mcp-registry-annotation-projection`: The fallback rule that projects every `server.json` attribute without a native home into `modelcontextprotocol.io/` annotations — dot-path encoding of nested objects and array indices, catalog-valid key sanitization/truncation, no-overwrite of natively-mapped fields, and scalar round-trip fidelity. + +### Modified Capabilities + +_(none — no long-lived specs exist under `openspec/specs/` yet; this change introduces new capabilities only and consumes the upstream Backstage `mcp-server` `API` entity contract defined in [`backstage/backstage`](https://github.com/backstage/backstage).)_ + +## Non-goals + +- **Ingestion / runtime.** No registry HTTP client, polling schedule, catalog entity provider/processor, or entity lifecycle management. This change defines the pure `server.json` → entity transform only; the component that fetches entries and applies the mapping is a separate future change. +- **Modifying the upstream `mcp-server` `API` entity contract.** This change consumes that contract as defined in [`backstage/backstage`](https://github.com/backstage/backstage); it does not change `spec.type: mcp-server` recognition, validation, or page rendering. Changes to the entity contract are owned upstream (Backstage RFC [#32062](https://github.com/backstage/backstage/issues/32062)), not made here. +- **Reverse mapping** (entity → `server.json`) beyond the scalar round-trip fidelity guarantee needed to avoid data loss. +- **Runtime invocation / health checking** of the mapped MCP servers. +- **Local package execution.** `packages[]` runtime details are preserved as annotations for discoverability, not interpreted or executed. +- **Deduplication / merge** of the same server appearing in multiple registries. + +## Upstream References + +The mapping target — the `mcp-server` `API` entity shape — is anchored to the following upstream Backstage sources (`backstage/backstage`): + +- **Canonical example** — [`packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml): the reference `mcp-server` `API` entity. It uses `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, `metadata.tags: [mcp, ai]`, and — notably — a **top-level `spec.remotes[]`** with `type` + `url` (no `spec.definition`): + ```yaml + apiVersion: backstage.io/v1alpha1 + kind: API + metadata: + name: backstage-mcp-server + description: An MCP server that exposes tools related to the Backstage ecosystem + tags: [mcp, ai] + spec: + type: mcp-server + lifecycle: experimental + owner: team-a + remotes: + - type: streamable-http + url: http://localhost:7007/api/mcp/v1 + ``` +- **Base API schema** — [`packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json): defines `spec.type` (free-form string), `lifecycle`, `owner`, and lists `definition` as required for generic `API` entities. This `definition` requirement is **overridden for `spec.type: mcp-server`** by the dedicated mcp-server entity schema below. +- **RFC** — Backstage [#32062](https://github.com/backstage/backstage/issues/32062): models MCP servers as the `API` kind with `spec.type: mcp-server`. +- **Dedicated mcp-server entity schema** — [`packages/catalog-model/src/kinds/McpServerApiEntity.ts`](https://github.com/backstage/backstage/blob/f91434377dc43cd64bef82344e3f2b539bfdaf11/packages/catalog-model/src/kinds/McpServerApiEntity.ts#L28-L36) (Backstage [PR #34016](https://github.com/backstage/backstage/pull/34016)): the upstream schema for `spec.type: mcp-server`. It does `Omit` and requires `spec.type: 'mcp-server'`, `spec.lifecycle`, `spec.owner`, and `spec.remotes[]` (with optional `spec.system`); `spec.definition` is not part of this schema. `spec.remotes` therefore **replaces** the base schema's required `spec.definition` for mcp-server entities. + +**Consequence for this mapping:** the mapping target is the upstream example shape exactly — `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, `metadata` (`name`/`title`/`description`/`tags`/`links`), and top-level `spec.remotes[]` (`type`, `url`). No `spec.definition` is emitted. `server.json` fields with no native home (`packages`, `repository` sub-fields other than `url`, `icons`, remote `headers`/`variables`, `_meta`, …) are projected into `modelcontextprotocol.io/*` annotations per `mcp-registry-annotation-projection`. (`name` and `version` get dedicated `modelcontextprotocol.io/name` and `modelcontextprotocol.io/version` fields and also form `metadata.name`.) + +## Canonical Touchpoints + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None (new capabilities only; `openspec/specs/` does not yet exist) + +**Change type**: feature-spec + +## Impact + +- **Depends on the upstream Backstage `mcp-server` `API` entity specification** (defined in [`backstage/backstage`](https://github.com/backstage/backstage), not by RHDH): the mapping targets that `mcp-server` `API` entity contract — `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, and top-level `spec.remotes[]` (`type`, `url`); no `spec.definition`. Produced entities MUST pass the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`, PR [#34016](https://github.com/backstage/backstage/pull/34016)), which for `spec.type: mcp-server` requires `spec.remotes` and drops the base schema's `spec.definition` requirement. +- **Source schema**: MCP Registry draft `server.schema.json` — top-level `name`, `title`, `description`, `version`, `websiteUrl`, `icons[]`, `repository{url,source,id,subfolder}`, `packages[]` (with nested `runtimeArguments`/`packageArguments`/`environmentVariables`/`transport`), `remotes[]` (`type`, `url`, `headers[]`, `variables`), and `_meta`. The draft schema evolves; the mapping is versioned against the draft and projects unknown fields generically (fail-open). +- **Backstage constraints**: annotation keys allow exactly one `/`; the name segment is ≤63 chars over a restricted character set — driving the dot-path encoding, sanitization, and hash-suffix truncation rules. +- **Consumers**: a future registry entity provider; AI agents and developers who discover MCP servers via catalog search/filter (the `modelcontextprotocol.io/*` annotations become searchable/filterable metadata). +- **Documentation**: mapping-table reference, worked examples, and a conformance fixture set usable as the test oracle when the transform is implemented. +- **Upstream**: keep aligned with Backstage RFC [#32062](https://github.com/backstage/backstage/issues/32062) and the MCP Registry `server.json` draft as both evolve. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md new file mode 100644 index 00000000000..524595c856c --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md @@ -0,0 +1,113 @@ +## MCP Registry Annotation Projection + +This capability defines the fallback that captures every `server.json` attribute which has no native home in the `mcp-server` `API` entity shape (see `mcp-registry-server-mapping`). Such attributes are projected into entity annotations under the `modelcontextprotocol.io/` prefix, keyed by the attribute's path within the source document. + +Attribute paths are encoded in **dot-separated** form as `modelcontextprotocol.io/attribute.tree.to.leaf`. Because Backstage annotation keys permit exactly one `/` (separating the DNS-style prefix from the name segment) and limit the name segment to a restricted character set and 63 characters, the name segment is sanitized and length-bounded so every produced key is catalog-valid. + +--- + +## ADDED Requirements + +### Requirement: Project unmapped server.json attributes into modelcontextprotocol.io annotations + +Every scalar leaf in the `server.json` document that is not consumed by a native field mapping SHALL be projected into an entity annotation whose key is `modelcontextprotocol.io/`, where `` identifies the attribute's location in the source document. Attributes that the direct mapping already places in a native field or a dedicated annotation SHALL NOT be re-projected by this fallback. + +#### Scenario: Unmapped scalar becomes an annotation + +- **WHEN** a `server.json` carries a scalar with no native home, such as `icons[0].mimeType: image/png` +- **THEN** the entity has an annotation `modelcontextprotocol.io/icons.0.mimeType` with value `image/png` + +#### Scenario: Natively-mapped attributes are not re-projected + +- **WHEN** a `server.json` carries `remotes[].type`/`url` (mapped to `spec.remotes`), `name` (mapped to `modelcontextprotocol.io/name` and `metadata.name`), `version` (mapped to `modelcontextprotocol.io/version`), `title`/`description` (mapped to `metadata`), and `websiteUrl`/`repository.url` (mapped to `metadata.links`, with `repository.url` also to the `backstage.io/source-location` annotation) +- **THEN** those attributes are not additionally emitted as generic `modelcontextprotocol.io/*` projected annotations, and the generic projection does not overwrite or re-derive the direct-mapping `backstage.io/source-location` annotation + +### Requirement: Encode nested paths as dot-separated segments within a single-slash key + +Nested object keys and array indices SHALL be encoded as dot-separated segments in the annotation key's name portion, appended to the single `modelcontextprotocol.io/` prefix. Object keys contribute their key name; array elements contribute their zero-based numeric index. Only scalar leaves are emitted as annotation values, serialized as strings; container nodes (objects, arrays) are traversed rather than emitted. + +#### Scenario: Nested object key path + +- **WHEN** a `server.json` carries `repository.source: github` +- **THEN** the entity has an annotation `modelcontextprotocol.io/repository.source` with value `github` + +#### Scenario: Array element index path + +- **WHEN** a `server.json` carries `packages[0].identifier: "@scope/pkg"` +- **THEN** the entity has an annotation `modelcontextprotocol.io/packages.0.identifier` with value `@scope/pkg` + +#### Scenario: Deeply nested leaf under an array + +- **WHEN** a `server.json` carries `remotes[1].headers[0].name: Authorization` +- **THEN** the entity has an annotation `modelcontextprotocol.io/remotes.1.headers.0.name` with value `Authorization` + +#### Scenario: Scalar values are serialized as strings + +- **WHEN** a projected leaf is a non-string scalar such as a boolean or number (e.g. `packages[0].environmentVariables[0].isSecret: true`) +- **THEN** the annotation value is the string form of that scalar (e.g. `true`) + +### Requirement: Produce catalog-valid annotation keys + +Every projected annotation key SHALL be valid for the Backstage catalog: the name segment SHALL contain only allowed characters (alphanumerics plus `-`, `_`, `.`), SHALL begin and end with an alphanumeric character, and SHALL be at most 63 characters. Path segments containing characters outside this set (for example `/`, `$`, `@`, whitespace, or a leading `_` as in `_meta`) SHALL be sanitized deterministically, and keys whose name segment would exceed 63 characters SHALL be truncated and suffixed with a stable hash so they remain valid and unique. + +#### Scenario: Illegal characters in a path are sanitized + +- **WHEN** a `server.json` carries a `_meta` object whose nested key contains a `/` (e.g. `_meta."io.modelcontextprotocol.registry/publisher-provided".x`) +- **THEN** the projected annotation key replaces the leading underscore and the embedded `/` with allowed characters so the resulting `modelcontextprotocol.io/` key is catalog-valid + +#### Scenario: Over-length key is truncated with a stable suffix + +- **WHEN** a projected path's name segment would exceed 63 characters +- **THEN** the mapping truncates it and appends a deterministic hash suffix derived from the full path, keeping the key ≤63 characters + +#### Scenario: Sanitization collisions are disambiguated + +- **WHEN** two distinct source paths sanitize to the same annotation key +- **THEN** the mapping appends a deterministic hash suffix so each source path maps to a distinct key + +### Requirement: Do not overwrite reserved or previously-set annotations + +Projection SHALL NOT overwrite annotations set by the direct mapping (for example `modelcontextprotocol.io/name`, `modelcontextprotocol.io/version`) or any other reserved annotation. If a generic projected key would collide with such an annotation, the direct-mapping value SHALL win and the projection SHALL be skipped or disambiguated. + +#### Scenario: Direct-mapping annotation wins + +- **WHEN** a generic projection would produce a `modelcontextprotocol.io/name` key that collides with the canonical-name annotation set by the direct mapping +- **THEN** the direct-mapping value is retained and the generic projection does not overwrite it + +### Requirement: Redact secret-flagged input values + +An `Input` object in `server.json` (as used by `packages[].environmentVariables[]`, `remotes[].headers[]`, `remotes[].variables`, and package/runtime arguments) MAY declare `isSecret: true`. When an input object declares `isSecret: true`, the projection SHALL prune (omit) that object's `default` and `value` leaves — those values SHALL NOT appear in any `modelcontextprotocol.io/*` annotation. All non-secret sibling leaves of the same input (for example `name`, `description`, `format`, `isRequired`, `isSecret`, `choices`) SHALL continue to project normally. This redaction applies uniformly to every `isSecret: true` input regardless of location, not only environment variables. + +#### Scenario: Secret environment variable value is pruned + +- **WHEN** a `server.json` carries `packages[0].environmentVariables[0]` with `isSecret: true` and a populated `default` (or `value`) +- **THEN** no `modelcontextprotocol.io/*` annotation carries that `default`/`value`, while the input's non-secret leaves (e.g. `packages.0.environmentVariables.0.name`, `.isSecret`, `.description`) are still projected + +#### Scenario: Secret remote header/variable value is pruned + +- **WHEN** a `remotes[].headers[]` or `remotes[].variables` input declares `isSecret: true` with a populated `default`/`value` +- **THEN** that `default`/`value` is omitted from all annotations, and the redaction behaves identically to the environment-variable case (uniform across input locations) + +#### Scenario: Non-secret input value is retained + +- **WHEN** an input object has `isSecret: false` or omits `isSecret`, with a populated `default`/`value` +- **THEN** that `default`/`value` is projected into a `modelcontextprotocol.io/*` annotation as normal + +### Requirement: Scalar round-trip fidelity + +Every scalar leaf present in the source `server.json` SHALL be recoverable from the produced entity — either from a native field or from a projected annotation — **except** the `default`/`value` leaves of `isSecret: true` inputs, which are intentionally redacted per "Redact secret-flagged input values". Null values and empty containers MAY be omitted per a documented rule; every non-null, non-redacted scalar SHALL be represented. + +#### Scenario: All scalar leaves are recoverable + +- **WHEN** a `server.json` with populated `packages`, `repository`, `icons`, and `_meta` is mapped +- **THEN** every non-null, non-redacted scalar leaf from those sections is present either in a native entity field or in a `modelcontextprotocol.io/*` annotation, so the source values can be reconstructed + +#### Scenario: Redacted secret leaves are exempt from round-trip + +- **WHEN** a `server.json` carries an `isSecret: true` input with a populated `default`/`value` +- **THEN** the absence of that `default`/`value` from the entity does NOT violate round-trip fidelity, because secret redaction is a documented exception + +#### Scenario: Nulls and empty containers follow the documented omission rule + +- **WHEN** a `server.json` attribute is `null` or an empty array/object +- **THEN** it is omitted from the annotations per the documented rule, and its omission does not cause the mapping to fail diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md new file mode 100644 index 00000000000..52f09a26895 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md @@ -0,0 +1,115 @@ +## MCP Registry Server Mapping + +This capability defines the deterministic, idempotent transform from a single MCP registry `server.json` document (conforming to the draft [`server.schema.json`](https://raw.githubusercontent.com/modelcontextprotocol/registry/refs/heads/main/docs/reference/server-json/draft/server.schema.json)) into a single Backstage `API` entity with `spec.type: mcp-server`. + +The target entity shape follows the upstream Backstage mcp-server example ([`backstage-mcp-server-api.yaml`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml)): `apiVersion: backstage.io/v1alpha1`, `kind: API`, `metadata` (`name`, `title`, `description`, `tags`, `links`, `annotations`), and `spec` (`type: mcp-server`, `lifecycle`, `owner`, top-level `remotes[]` with `type` + `url`). No `spec.definition` is emitted — top-level `spec.remotes[]` replaces it. + +`server.json` attributes without a native home in this shape are handed to `mcp-registry-annotation-projection`. This spec covers only the direct (native-field) mapping and the supply of catalog-required fields absent from the source. + +--- + +## ADDED Requirements + +### Requirement: Transform a server.json document into a valid mcp-server API entity + +The mapping SHALL transform one `server.json` document into one `API` entity with `apiVersion: backstage.io/v1alpha1`, `kind: API`, and `spec.type: mcp-server`, such that the produced entity passes the upstream Backstage `mcp-server` `API` entity schema (`McpServerApiEntity`), which for `spec.type: mcp-server` requires `spec.remotes` and does not include `spec.definition` — overriding the generic base `API` schema. When the source omits a `server.json`-required field (`name`, `description`, or `version`), the mapping SHALL fail with an actionable error that names the missing field and references the MCP server schema, rather than emitting a partial entity. + +#### Scenario: Minimal valid server.json produces a valid entity + +- **WHEN** the mapping receives a `server.json` with `name`, `description`, `version`, and one `remotes` entry (`type: streamable-http`, valid `url`) +- **THEN** it produces an `API` entity with `kind: API`, `spec.type: mcp-server`, `spec.lifecycle` (`production` when no caller override is supplied), `spec.owner` (`unknown` when no caller override is supplied), and a top-level `spec.remotes` entry carrying that `type` and `url`, and the entity passes the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`) validation + +#### Scenario: Missing required source field fails the mapping + +- **WHEN** the input `server.json` omits a required field such as `name`, `description`, or `version` +- **THEN** the mapping fails with an error that names the missing field and references the MCP server schema, and no entity is produced + +### Requirement: Map remotes to top-level spec.remotes + +The mapping SHALL copy each `server.json` `remotes[]` entry's `type` and `url` into a corresponding top-level `spec.remotes[]` entry on the `API` entity, preserving source order. When the source `remotes` is empty or unset, the mapping SHALL emit an empty `spec.remotes: []` array — never an omitted field — so the output stays deterministic and schema-conformant. The mapping SHALL NOT emit a `spec.definition` field. Remote sub-fields that are not part of the native `spec.remotes` shape (`headers`, `variables`) SHALL be handed to `mcp-registry-annotation-projection` rather than dropped. + +#### Scenario: Remotes copied in order + +- **WHEN** a `server.json` declares multiple `remotes` entries +- **THEN** `spec.remotes` contains one entry per source remote, in the same order, each with the source `type` and `url`, and no `spec.definition` is present + +#### Scenario: Remote headers and variables are projected, not dropped + +- **WHEN** a `remotes` entry carries `headers` or `variables` +- **THEN** the native `spec.remotes` entry contains only `type` and `url`, and the `headers`/`variables` are projected into `modelcontextprotocol.io/*` annotations keyed by the remote's index + +#### Scenario: Server with no remotes + +- **WHEN** a `server.json` declares no `remotes` (only local `packages`) +- **THEN** the entity is still produced with an empty `spec.remotes: []` (not omitted), remains valid, and the `packages` are projected into annotations + +### Requirement: Derive a version-unique metadata.name and preserve the canonical name and version + +A registry publishes one `server.json` per server version, and each version becomes its own `API` entity; a `metadata.name` derived from the canonical name alone would therefore collide across versions in the catalog. The `server.json` `name` is also a reverse-DNS identifier (`namespace/server`) that is not itself a valid Backstage `metadata.name`. The mapping SHALL derive `metadata.name` as `__` — the sanitized canonical name and the sanitized version joined by a double underscore (`__`) — conforming to the Backstage name character set (lowercase alphanumerics with `-`/`_`/`.`, beginning and ending alphanumeric, ≤63 characters). The mapping SHALL preserve the unmodified canonical name (without the version) in a `modelcontextprotocol.io/name` annotation, and SHALL map the `version` directly to its dedicated `modelcontextprotocol.io/version` annotation so it remains individually queryable. + +#### Scenario: Two versions of the same server produce distinct entities + +- **WHEN** two `server.json` documents share the canonical name `io.github.user/weather` but declare `version` `1.0.0` and `2.0.0` +- **THEN** the two produced entities have distinct `metadata.name` values (each incorporating its version), each carries the same `modelcontextprotocol.io/name: io.github.user/weather`, and each carries its own `modelcontextprotocol.io/version` (`1.0.0` and `2.0.0` respectively) + +#### Scenario: Reverse-DNS name and version are sanitized into metadata.name + +- **WHEN** `server.json` `name` is `io.github.user/weather` and `version` is `1.0.2` +- **THEN** `metadata.name` is the sanitized `__` form (e.g. `io.github.user-weather__1.0.2`), the original `io.github.user/weather` is preserved verbatim in `modelcontextprotocol.io/name`, and `1.0.2` is recorded in `modelcontextprotocol.io/version` + +#### Scenario: Over-length or colliding names remain unique + +- **WHEN** two distinct (name, version) pairs sanitize to the same `__`, or the combined value exceeds the 63-character limit after sanitization +- **THEN** the mapping produces a deterministic, unique `metadata.name` by truncating and appending a stable hash suffix derived from the canonical name and version + +### Requirement: Map descriptive metadata to native Backstage fields + +The mapping SHALL map `server.json` descriptive attributes to native Backstage `metadata` fields: `title` → `metadata.title`; `description` → `metadata.description`; `websiteUrl` → a `metadata.links` entry whose `url` is the `websiteUrl` and whose `title` is `Website`; and `repository.url` (combined with `repository.subfolder` when present) → a `metadata.links` entry whose `url` references the repository and whose `title` is `Source Code`. The mapping SHALL ALSO emit `repository.url` (combined with `repository.subfolder` when present) as a `backstage.io/source-location` annotation, so the repository is captured both as the canonical Backstage source-location annotation (for source-aware tooling) and as a human-visible source link. The mapping SHALL set `metadata.tags` to include the upstream mcp-server convention tags (`mcp`, `ai`). + +#### Scenario: Descriptive fields lift to metadata + +- **WHEN** a `server.json` provides `title`, `description`, and `websiteUrl` +- **THEN** the entity has `metadata.title` from `title`, `metadata.description` from `description`, and a `metadata.links` entry whose `url` is `websiteUrl` and whose `title` is `Website` + +#### Scenario: Repository maps to a source link and a source-location annotation + +- **WHEN** a `server.json` provides `repository.url` and `repository.subfolder` +- **THEN** the entity has a `metadata.links` entry whose `url` references the repository URL (and subfolder when present) and whose `title` is `Source Code`, a `backstage.io/source-location` annotation carrying the same repository URL (and subfolder when present) is also emitted, and the remaining `repository` sub-fields (`source`, `id`) are projected into `modelcontextprotocol.io/*` annotations + +#### Scenario: mcp-server tags applied + +- **WHEN** any `server.json` is mapped +- **THEN** `metadata.tags` includes `mcp` and `ai` + +### Requirement: Supply catalog-required fields absent from server.json + +`server.json` does not carry a Backstage owner or lifecycle. The mapping SHALL set `spec.owner` to the constant `unknown` when no caller override is supplied, and to a caller-provided default when one is supplied; the mapping SHALL NOT fail for a missing owner. The mapping SHALL set `spec.lifecycle` to the constant `production` when no caller override is supplied, and to a caller-provided default lifecycle when one is supplied; the mapping SHALL NOT fail for a missing lifecycle. + +#### Scenario: Owner defaults to unknown + +- **WHEN** the mapping is invoked without a caller-provided owner and `server.json` carries no owner information +- **THEN** `spec.owner` is set to `unknown` and the mapping succeeds + +#### Scenario: Owner override supplied by caller + +- **WHEN** the mapping is invoked with a caller-provided default owner +- **THEN** `spec.owner` is set to that owner value + +#### Scenario: Lifecycle defaults to production + +- **WHEN** the mapping is invoked without a caller-provided lifecycle +- **THEN** `spec.lifecycle` is set to `production` and the mapping succeeds + +#### Scenario: Lifecycle override supplied by caller + +- **WHEN** the mapping is invoked with a caller-provided default lifecycle +- **THEN** `spec.lifecycle` is set to that lifecycle value + +### Requirement: Deterministic and idempotent mapping + +The mapping SHALL be a pure function of its inputs (the `server.json` document and the caller-provided defaults): given identical inputs it SHALL produce a byte-identical entity, with stable ordering of `spec.remotes`, `metadata.tags`, and annotation keys, and SHALL NOT introduce timestamps, random values, or ingestion-source state. + +#### Scenario: Same input yields identical output + +- **WHEN** the mapping is run twice on identical inputs +- **THEN** the two produced entities are identical, including annotation key ordering and `spec.remotes` ordering diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/tasks.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/tasks.md new file mode 100644 index 00000000000..966cad0e4e8 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/tasks.md @@ -0,0 +1,37 @@ + + + +## 1. Mapping Reference & Schema Pinning + +- [ ] 1.1 Pin the source `server.json` draft schema version this mapping targets and record it (with URL and retrieval date) in a `mapping-reference.md` under the change +- [ ] 1.2 Author the canonical field-mapping table in `mapping-reference.md`: each `server.json` attribute → native entity target OR "projected annotation", including `name`→`metadata.name`(`__`)+`modelcontextprotocol.io/name`, `version`→`modelcontextprotocol.io/version`, `description`/`title`, `websiteUrl`→`metadata.links` (title `Website`), `repository.url`→`metadata.links` (title `Source Code`) + `backstage.io/source-location`, `tags`, `remotes[]`→`spec.remotes[]` +- [ ] 1.3 Document the annotation key rules (dot-separated `modelcontextprotocol.io/attribute.tree.to.leaf`, character sanitization, 63-char truncation + stable hash suffix, collision disambiguation) with worked examples +- [ ] 1.4 Document field-supply rules (`owner` defaults to the constant `unknown`, `lifecycle` defaults to the constant `production` — both overridable by caller defaults, never a failure) and the null/empty-container omission rule + +## 2. Direct Field Mapping Implementation + +- [ ] 2.1 Implement the `server.json` → `mcp-server` `API` entity transform skeleton (pure function of the document plus caller defaults; no I/O, no timestamps, no randomness) +- [ ] 2.2 Implement identity derivation: sanitize and combine `__` for `metadata.name`, preserve bare name in `modelcontextprotocol.io/name`, version in `modelcontextprotocol.io/version`, with truncation + hash-suffix fallback +- [ ] 2.3 Implement descriptive-metadata mapping (`title`, `description`, `websiteUrl`→`metadata.links` entry titled `Website`, `repository.url` (combined with `repository.subfolder` when present)→`metadata.links` entry titled `Source Code` **and** a `backstage.io/source-location` annotation, `tags` including `mcp`/`ai`) +- [ ] 2.4 Implement `remotes[]` → top-level `spec.remotes[]` (`type`, `url`, source order preserved); when the source has no `remotes` (empty or unset), emit an empty `spec.remotes: []` array (never omitted); ensure no `spec.definition` is emitted +- [ ] 2.5 Implement field-supply: `spec.type: mcp-server`, `spec.owner` set to `unknown` by default (caller override allowed, never a failure), `spec.lifecycle` set to `production` by default (caller override allowed, never a failure) +- [ ] 2.6 Enforce required-source-field validation (`name`, `description`, `version`) with actionable errors that name the missing field + +## 3. Annotation Projection Implementation + +- [ ] 3.1 Implement the recursive scalar-leaf walker over `server.json` that builds dot-separated paths (object keys, zero-based array indices) +- [ ] 3.2 Implement key sanitization (illegal chars incl. `/`, `$`, `@`, whitespace, leading `_`), the ≤63-char truncation + stable hash suffix, and sanitization-collision disambiguation +- [ ] 3.3 Implement value serialization (scalars to strings) and the null/empty-container omission rule +- [ ] 3.4 Implement the no-overwrite guard so generic projection never clobbers native/dedicated/reserved annotations (`modelcontextprotocol.io/name`, `/version`, …) and skips all source fields with native mappings per `mcp-registry-server-mapping` (e.g. `name`, `version`, `title`, `description`, `websiteUrl`, `repository.url`, `remotes[].type`/`url`) +- [ ] 3.5 Wire remote sub-fields (`headers`, `variables`) and non-native sections (`packages`, `icons`, `repository.source`/`id`, `_meta`) through projection so nothing is dropped +- [ ] 3.6 Implement secret redaction (D9): when an `Input` object declares `isSecret: true`, prune its `default`/`value` leaves from projection (uniformly across `environmentVariables`, remote `headers`/`variables`, and arguments) while still projecting the input's non-secret leaves + +## 4. Conformance Fixtures, Verification & Docs + +- [ ] 4.1 Create input→expected-output fixtures: minimal server, multi-version (same name, two versions → distinct entities), server with `packages`/`icons`/`_meta`, server with remote `headers`/`variables`, and over-length/collision name cases +- [ ] 4.2 Verify every produced entity passes the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`, PR #34016) — `spec.remotes` required, `spec.definition` not required — rather than the generic base `API` schema +- [ ] 4.3 Verify determinism/idempotency (byte-identical output on repeated runs) and scalar round-trip fidelity (every non-null, non-redacted source scalar recoverable — `isSecret: true` `default`/`value` leaves exempt) via tests over the fixtures +- [ ] 4.4 Add unit tests covering annotation key sanitization, truncation, and collision disambiguation edge cases +- [ ] 4.5 Write the user/consumer documentation: the mapping guide, annotation-key conventions, and a worked `server.json`→entity example (aligned with the upstream `backstage-mcp-server-api.yaml` shape) +- [ ] 4.6 Add a fixture and test covering secret redaction (D9): an `isSecret: true` env var / remote header with a populated `default`/`value` produces no annotation carrying that value, while non-secret sibling leaves still project