lint: no reference-integrity for AI metadata (agent skills, skill tools) — R7, blocked on D1/D2

[os-zhuang]

Split out of #3583, which is otherwise closed: eight of its nine bug categories are now gated on validate/lint/compile, and the ninth (hook body write sets) is an accepted, documented gap. This is the one category with resolvable references and no rule — assessment R7 in docs/audits/2026-07-app-metadata-reference-integrity-assessment.md §3.

The gap

packages/lint/src contains zero AI rules. Nothing checks that:

• agent.skills[] names a skill the stack declares (ai/agent.zod.ts)
• skill.tools[] — wildcard-aware — names a declared tool (ai/skill.zod.ts)
• agent.tools[].name resolves to an action/flow by its declared type

Evidence on the real corpus (2026-07-28, hotcrm @ 8b28fa2, v2.2.2)

Surface	Count
agents	2
skills	6
agent.skills references	8
skill.tools references	16
tools declared by the stack	0

So all 16 skill→tool references are dead today — the original audit's "agent skills listing 11 tools with no definitions", still shipping. Every one passes objectstack validate and objectstack lint cleanly.

Two decisions come first (neither is a lint question)

• D2 — runtime-registered kernel skills/tools vs. static lint. The ask/build kernel agents register their skills/tools via service at boot, so a stack can legitimately reference names no static analysis can see (same invisibility as plugin objects, which feat(lint): reference-integrity validation for object and action names (#3583) #3657 solved with the curated PLATFORM_PROVIDED_OBJECT_NAMES registry). Either extend that precedent with official skill/tool names, or run R7 at advisory severity. Small; take it when the rule starts.
• D1 — agent.knowledge.indexes has no definition site. KnowledgeSourceSchema is never referenced by stack.zod.ts, so the namespace is unresolvable by construction — parsed, unmarked, and unenforceable, which is the state ADR-0049's trichotomy prohibits. Either add a knowledgeSources slot to the stack (spec change, ADR-worthy) or mark the fields [EXPERIMENTAL — not enforced]. Linting a namespace with no definition site would institutionalise the gap, so the knowledge half of R7 stays blocked until this lands. HotCRM carries 2 such references.

Out of scope

The "hand-off to a nonexistent skill" instance from the original audit: hand-off has no schema field anywhere — it lives in free-form prompt text, which has no structure to check (§7 non-goal).

Done looks like

validate-ai-references in @objectstack/lint, appended to REFERENCE_INTEGRITY_RULES (one line — validate/lint/compile pick it up automatically), with fixtures from the HotCRM instances and a zero-false-positive pass over examples/. Note the verification lesson from #3806: the false-positive floor must be read per branch — an example corpus that declares no agents proves nothing about the agent branch.

Refs #3583 · ADR-0049 · ADR-0072 · ADR-0078

[os-zhuang]

Positioning check before D1/D2: the agent branch lints a surface ADR-0063 closed, and the tool branch has no registry to resolve against

Read of the three repos at today's HEAD (objectstack @ 87aca93, cloud, hotcrm). The conclusion is that R7 should not be built as specced — not because the references don't matter, but because two of its three branches are checking namespaces whose authoring status is itself unresolved. There is a decision upstream of D1/D2.

1. Agents are already a closed surface — but only half-closed, and not at the authoring layer

ADR-0063 §2 decided it: "third parties author *.skill.ts and tools / MCP connectors. *.agent.ts is closed to third parties", and ADR-0040 §3 (tenant custom agents) is withdrawn. That landed in two places:

• spec/src/kernel/metadata-plugin.zod.ts:697 — agent carries allowRuntimeCreate: false, allowOrgOverride: false.
• cloud/packages/service-ai/src/agent-runtime.ts:199-208 — listAgents() filters the catalog to platform-owned records.

It did not land at the authoring layer, and the withdrawal has holes:

	state
stack.agents slot	still open — spec/src/stack.zod.ts:303
stack.zod.ts:295-298 doc	still describes the withdrawn ADR-0040 model ("Persona-bearing copilots (1-3 per app)… the active app's defaultAgent is selected automatically")
app.defaultAgent	still in spec (ui/app.zod.ts:675), still an editable field in ui/app.form.ts:39, and its doc example is literally defaultAgent: 'sales_copilot' — the withdrawn pattern
loadAgent() (chat route)	no platform filter — agent-runtime.ts:244
resolveDefaultAgent()	reads app.defaultAgent and loads it unfiltered — agent-runtime.ts:496-502

So a stack-authored agent today is hidden from the picker but still reachable — via POST /agents/:name/chat and via app.defaultAgent. HotCRM's two agents are exactly that state: parsed, built into the artifact, invisible, reachable.

This is the ADR-0078 prohibited state, and the remedy ADR-0078 prescribes is not a lint rule about the slot's contents — it is closing or marking the slot. Writing validate-ai-references over agent.skills[] would institutionalise an authoring surface the platform decided not to have — the same objection the issue itself raises for the knowledge half in D1.

Corroborating: on the motivating corpus the agent branch finds nothing. HotCRM's 8 agent.skills references (sales_copilot 5, service_copilot 3) all resolve against its 6 declared skills. 8/8 clean.

2. stack.tools has zero readers — the tool half cannot resolve because there is no tool authoring story yet

Searched both repos for any read of the tool metadata type:

grep -rE "(list|get|register|load)\(\s*'tool'" objectstack/packages cloud/packages
→ cloud/packages/service-ai-studio/src/plugin.ts:538   register('tool', …)   ← write only

One write, no reads. The plugin's own comment says it: "Studio visibility: persist tool/agent/skill metadata… nothing subscribes to tool/agent/skill events" (plugin.ts:518-524). The executable path is the in-process ToolRegistry (aiService.toolRegistry.getAll(), routes/agent-routes.ts:582), which the metadata service never feeds.

Consequences:

• A tool declared in an app stack registers a record, never a callable — ToolSchema is pure declaration with no handler binding. "Third parties author tools" (ADR-0063 §2) is not currently possible; today the real third-party primitive is skills alone, and a skill's tools[] can only name runtime-registered tools it does not own.
• 'tools' is missing from CONCAT_ARRAY_FIELDS (stack.zod.ts:1161-1188, which does list 'agents' and 'skills'), so composeStacks silently drops it — consistent with having no consumer.
• plugin.ts:527-531 skips registration if (exists), so a tenant-declared tool of the same name would suppress the platform's own metadata mirror. Shared namespace, no prefixing.

3. Re-measuring the corpus: the "all 16 dead" figure is an artifact of resolving against the wrong registry

The issue reads 16 dead skill.tools references because the stack declares 0 tools. Resolved against the registry that actually decides the call — the in-process one — the split is:

	refs
resolve today (describe_object, list_objects, query_records, get_record, aggregate_data, search_knowledge)	6
genuinely fictional (triage_case, generate_email_copy, optimize_subject_line, personalize_content, generate_email, analyze_lead, suggest_next_action, analyze_pipeline, identify_at_risk, forecast_revenue)	10

A rule resolving against stack.tools alone would therefore emit 6 false positives out of 16 (37.5%) on its own motivating corpus — a rate that gets a rule disabled rather than fixed. And the runtime treats the miss as tolerable by design: "Tools that are declared by a skill but missing from the available tool registry are silently dropped — this is intentional so a skill can be authored before all its underlying tools are registered" (skill-registry.ts:196-199, same wording at agent-runtime.ts:376-379).

Also worth noting for the verification plan: examples/ declares zero agents, skills and tools (grep -rl "defineSkill|defineAgent|defineTool" examples/ → 0 files). Per the #3806 lesson the issue already cites, the false-positive floor for R7 is unmeasurable on examples/ — every branch, not just the agent one.

4. What is actually worth building — and it is not in R7's list

resolveActiveSkills() throws on a skill↔agent surface mismatch (agent-runtime.ts:468-477, ADR-0064 §3). That is a statically decidable, runtime-fatal condition surfacing as a 500 at chat time. Zero-false-positive, cheap, and it protects the invariant that keeps ask from authoring. This is the AI rule with the best evidence behind it, and the issue does not mention it.

Two adjacent findings from the same read, offered as separate work:

• ADR-0064's "no global fall-through" is not enforced at the agent.tools[] seam. buildRequestOptions() resolves agent.tools[].name against the full registry passed in (agent-runtime.ts:409-424), with no surface check — while the route passes toolRegistry.getAll(), which includes AI Studio's authoring tools when that plugin is loaded. Combined with §1's unfiltered loadAgent/app.defaultAgent, a stack-authored surface:'ask' agent can name build-surface tools. Whether that is exploitable depends on the per-tool authz interceptors, which I did not audit — but the scoping invariant ADR-0064 states is not enforced here.
• skill.permissions is silently stripped. HotCRM's live_data skill declares permissions: ['crm:read'] (hotcrm/src/skills/live-data.skill.ts); SkillSchema has no such field, so Zod drops it. A permission-shaped declaration that does nothing — ADR-0049 territory.

5. Proposed disposition

Insert D0 ahead of D1/D2: decide whether tool is a third-party-authorable metadata type or in-process registration only. Until that is answered, stack.tools is inert metadata and no reference rule over it can be correct in either direction. It is also the decision that determines whether ADR-0063 §2's "skills and tools" is a shipped contract or an aspiration.

Then, in dependency order:

1. Finish the ADR-0063 withdrawal instead of linting around it. Close stack.agents to app packages, or mark it [EXPERIMENTAL — INTERNAL] per ADR-0078; fix the unfiltered loadAgent/resolveDefaultAgent paths; retire or re-document app.defaultAgent and the stale stack.zod.ts:295-298 prose. HotCRM's two agents then get deleted or become legitimate — either way the R7 agent branch has nothing left to check.
2. Ship the surface-affinity rule (§4) — it stands on its own regardless of D0/D1/D2.
3. R7's skill.tools branch, only after D0, at advisory severity (S2) and resolving against a published tool-name registry. Feasible: the platform already publishes its tool names to the metadata service at boot (plugin.ts:538), so the D2 choice is between snapshotting that into a PLATFORM_PROVIDED_TOOL_NAMES set (the PLATFORM_PROVIDED_OBJECT_NAMES / feat(lint): reference-integrity validation for object and action names (#3583) #3657 precedent) or staying advisory. Without one of those, the 6/16 false-positive rate above stands.
4. D1 — agree with the issue, take the marker, not the slot. Add [EXPERIMENTAL — not enforced] to agent.knowledge.indexes/sources. Adding a knowledgeSources stack slot would create a second namespace with no reader, i.e. the §2 problem again.

Net: R7 as scoped is ~M of work that would find 10 real problems and 6 false ones, over two namespaces whose authoring status the platform has not settled. Settling D0 and finishing the §1 withdrawal is smaller, removes the false positives by construction, and is a prerequisite either way.

---

Generated by Claude Code

[os-zhuang]

Disposition update (per maintainer direction): the knowledge half (D1) is deferred entirely — not a current priority, no marker work either. The rest of the assessment above is now in flight as three PRs:

• feat(lint,spec): AI surface-affinity rule, ADR-0063 doc completion, ADR-0109 draft (#3820) #3871 (this repo) — validate-ai-surface-affinity in REFERENCE_INTEGRITY_RULES (the statically provable, runtime-fatal AI check; zero FP by construction), the ADR-0063 doc-truth fixes (stack.agents marked platform-internal, app.defaultAgent re-documented as a surface-binding knob, SkillSchema documents that per-skill permissions doesn't exist), and ADR-0109 (Proposed) — the D0 decision this issue is blocked on: third-party tools are bindings to actions/flows; platform tool names become a conformance-tested registry. CI green.
• objectstack-ai/cloud#904 — closes the "hidden from the picker but still reachable" hole: loadAgent() now carries the same platform-ownership gate as listAgents(), so a withdrawn tenant agent 404s on direct chat and app.defaultAgent falls back to ask.
• fix(skills): drop the per-skill permissions keys — the field does not exist in SkillSchema hotcrm#511 — deletes the six dead skill.permissions keys (ADR-0049 shape).

R7's remaining buildable scope after this: the skill.tools branch only, blocked on ADR-0109 acceptance (which replaces D0/D2 here), warning-first per the ADR-0078 ratchet.

---

Generated by Claude Code