Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ jobs:
- run: npm run render -- --check
# Frontmatter, skill references, manifests against their JSON Schemas, and overrides.lock.
- run: npm run validate
# The mistakes render and validate must catch (scripts/test.mjs).
- run: npm test

workflows:
name: Lint workflows
Expand Down
23 changes: 18 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
# Contributing

Each host's plugin in `dist/<host>/` is rendered from shared skills plus that host's own files. Edit the sources, never `dist/`.
Each host's plugin in `dist/<host>/` is rendered from shared skills plus that host's own files. Code hosts (a piece for another project, a marketplace listing) are rendered the same way from their own source and the shared text. Edit the sources, never `dist/`.

## Layout

```
skills/<skill>/SKILL.md shared skills; {{var}} and {{> slot}} mark what differs per host
shared/assets/ files every host ships as they are (logo)
shared/assets/ files every plugin host ships as they are (logo)
shared/text/<name>.md one fact in plain text, used by skills ({{< name}}) and code hosts ({{text:name}})
platforms/<host>/
vars.yaml values for {{var}}
skills/<skill>/<slot>.md the host's text for a {{> slot}} in that skill
overrides.lock whole-file overrides and the shared version each was copied from (only once one exists)
everything else the host's manifests, README, …, copied into dist/<host>/ ({{version}} comes from release-please)
dist/<host>/ GENERATED: the plugin the host installs
dist/<host>/ GENERATED: the plugin the host installs, or a code host's files
.github/plugin/marketplace.json Copilot's marketplace catalog (see below)
scripts/ render, validate, override, new-host, mirror
scripts/ render, validate, override, new-host, mirror, test
```

### Marketplace catalogs
Expand All @@ -38,7 +39,19 @@ Requires Node 22. Run `npm ci` once.
- **A shared skill a host shouldn't ship:** list it under `skip_skills:` in that host's `vars.yaml` (Cursor uses a rules file instead of `using-zenrows`). A `{{skill:…}}` reference to a skipped skill fails the render.
- **A reference to another skill:** write `{{skill:batch}}`, never the plain name. It renders as that skill's name on the host: `skill_prefix` plus the shared name, unless `skill_names` in `vars.yaml` renames it (OpenClaw ships `scrape-webpage` as `zenrows-scrape`). The skill's own `name:` in its frontmatter uses it too.

Unknown variables, unknown skill names, missing slots and unused slot files fail the render, and `npm run validate` fails on a reference to a skill the host doesn't ship, and on a skill name written plainly instead of as `{{skill:…}}`.
Unknown variables, unknown skill names, missing slots, unused slot files, unknown snippets and any of these tokens left in a rendered file (a typo, or a file type the render doesn't fill) fail the render. Other `{{…}}`, such as an n8n or Make expression or JSX's `style={{…}}`, isn't one of these tokens and ships as written, and `npm run validate` fails on a reference to a skill the host doesn't ship, and on a skill name written plainly instead of as `{{skill:…}}`.

## Shared text

A fact that more than one place states (what an error code means, what a request costs) lives once in `shared/text/<name>.md`, as one line of plain text with no `{{…}}`. A skill or a text file inlines it with `{{< name}}`; code uses `{{text:name}}`, which renders as a quoted string, and adds its own words around it. Snippets stay plain text so they read right in both, and `npm run validate` fails on markdown in one (backticks, `*`, a link or a heading), or on one nothing uses.

## Code hosts

`kind: code` in `vars.yaml` makes a host whose output isn't a plugin but files for somewhere else: code for another project (a piece as it sits in that project's repo), or a listing a marketplace reads. A code host ships only its own files, with `{{…}}` filled in source files too: no shared skills, `LICENSE` or assets, and no `plugin.json`. Its `README.md` documents it here and isn't shipped or rendered; `npm run validate` requires one, and the render fails if the host has a `skills/` folder. It needs no release-please entry until it's released from this repo.

## Tests

`npm test` (CI runs it too) checks that render and validate catch the mistakes they're meant to: each case in `scripts/test.mjs` makes one change in a temporary copy of the repo (a throwaway code host, a snippet, …) and expects a given error, or a pass. When you add a rule, add a case that breaks it.

## Manifest schemas

Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"prepare": "git rev-parse --git-dir >/dev/null 2>&1 && git config core.hooksPath .githooks || true",
"render": "node scripts/render.mjs",
"validate": "node scripts/validate.mjs",
"test": "node --test scripts/test.mjs",
"override": "node scripts/override.mjs create",
"override:ack": "node scripts/override.mjs ack",
"new-host": "node scripts/new-host.mjs",
Expand Down
79 changes: 71 additions & 8 deletions scripts/lib.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,23 @@ export const root = fileURLToPath(new URL("..", import.meta.url));
export const skillsDir = join(root, "skills");
export const platformsDir = join(root, "platforms");
export const distDir = join(root, "dist");
export const sharedTextDir = join(root, "shared", "text");

// Files copied into every host's plugin as they are.
const sharedFiles = { LICENSE: "LICENSE" };
const sharedDirs = { "shared/assets": "assets" };
// In platforms/<host>/, these are inputs to the render, not files of the plugin.
// Everything else there (manifests, README, …) is copied into dist/<host>/.
const reserved = ["vars.yaml", "overrides.lock", "skills/"];
// What's left of one of this tool's tokens ({{name}}, {{skill:x}}, {{text:x}},
// {{< x}}, {{> x}}) after the render, for example in a file type it doesn't
// fill, or with a stray space, capital letter, _ or . the fill regexes don't
// take. A colon is ours only after skill or text, so JSX's style={{ flex: 1 }}
// isn't matched; nor is other tools' {{…}} (Make's {{a.b}}, n8n's {{ $json.x }}),
// which ships as written. scripts/test.mjs lists both sides.
export const leftoverToken = /\{\{\s*(?:[<>]\??\s*[a-z0-9_.-]+|(?:skill|text)\s*:\s*[a-z0-9_.-]+|[a-z_][a-z0-9_-]*)\s*\}\}/i;
// {{vars}} are filled in files with these extensions; others are copied as bytes.
const textFile = /\.(json|md|mdc|ya?ml|txt|toml)$/;
export const textFile = /\.(json|md|mdc|ya?ml|txt|toml|[cm]?[jt]s)$/;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allowlist fails quietly, which was fine when the output was manifests and markdown but is riskier now that a code host’s whole output is code. Anything with an extension that isn’t on the list gets copied byte for byte, placeholders and all, with no warning.

I set up a kind: code host to see the effect:

platforms/piece/comp.tsx → dist/piece/comp.tsx:  export const a = "{{host_full}}";
platforms/piece/setup.py → dist/piece/setup.py:  name = "{{host_full}}"

Both render and validate were clean. .tsx, .jsx, .vue, .py and .go all land here.

One option instead of extending the regex each time a host brings a new file type is to check the rendered output. Something like this at the end of renderHost, once out is built:

for (const [path, buf] of out) {
  const text = buf.toString("utf8");
  if (!Buffer.compare(Buffer.from(text, "utf8"), buf) && text.includes("{{")) {
    throw new Error(`${host}: ${path} still has an unrendered {{…}}`);
  }
}

It matches what CONTRIBUTING already describes (“Unknown variables … fail the render”), and it covers the {{ < name}} case above and any future variation of the same thing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took the idea of checking the output: renderHost now fails on any of this tool's tokens left in a rendered file ({{name}}, {{< x}}, {{text:x}}, …), whatever its extension, and the error names the leftover. I first made it fail on any {{, but that blocked JSX style={{…}} and Make/n8n expressions, which code hosts will need, so it only matches our own token shapes (9cb7a04). The allow-list still decides which files get filled. In #5 I'll reword the one doc comment that mentions {{text:…}} when it merges main.


// Errors here are for whoever edited a skill or platform file: print the
// message, not a stack trace.
Expand Down Expand Up @@ -52,9 +60,15 @@ function versions() {

export function vars(host) {
const v = parseYaml(readFileSync(join(platformsDir, host, "vars.yaml"), "utf8")) ?? {};
if (typeof v !== "object" || Array.isArray(v)) throw new Error(`platforms/${host}/vars.yaml must be a list of name: value lines`);
if ("kind" in v && v.kind !== "plugin" && v.kind !== "code") {
throw new Error(`platforms/${host}/vars.yaml: kind is ${JSON.stringify(v.kind)}; use plugin or code`);
}
const version = versions()[`dist/${host}`];
if (!version) throw new Error(`.release-please-manifest.json has no "dist/${host}" entry`);
return { ...v, version };
// A code host may not be released from here (yet), so it needs no
// release-please entry; it then has no {{version}}.
if (!version && !isCodeHost(v)) throw new Error(`.release-please-manifest.json has no "dist/${host}" entry`);
return version ? { ...v, version } : v;
}

// overrides.lock: "<shared path>: <sha256>" per line; # starts a comment.
Expand All @@ -68,6 +82,13 @@ export function readLock(host) {
// the host's own skills (platforms/<host>/skills/<name>/SKILL.md with no shared
// skill of that name). Returns [{ name, own }].
export function hostSkills(host, values = vars(host)) {
// A code host (kind: code) ships no skills, so a skills/ folder there would be ignored.
if (isCodeHost(values)) {
if (listFiles(join(platformsDir, host, "skills")).length) {
throw new Error(`platforms/${host}/skills/: a code host (kind: code) ships no skills; move these files out of skills/`);
}
return [];
}
const shared = skillNames();
const skip = values.skip_skills ?? [];
for (const name of skip) {
Expand All @@ -83,22 +104,52 @@ export function hostSkills(host, values = vars(host)) {
].sort((a, b) => a.name.localeCompare(b.name));
}

// kind: code in vars.yaml marks a host whose output isn't a plugin but files for
// somewhere else (code for another project, a listing a marketplace reads): it
// ships only its own files, with {{…}} filled in, and no shared skills, LICENSE
// or assets.
export function isCodeHost(values) {
// vars() has already rejected any other kind, naming the file.
return values.kind === "code";
}

// A skill's name on a host: skill_prefix + the shared name, unless vars.yaml
// renames it in skill_names (OpenClaw ships scrape-webpage as zenrows-scrape).
export function hostSkillName(values, name) {
return values.skill_names?.[name] ?? `${values.skill_prefix ?? ""}${name}`;
}

// {{name}} is a variable from vars.yaml; {{skill:name}} is that skill's name on the host.
// shared/text/<name>.md: one fact in plain text, used by skills and by code
// hosts alike, so a wording fix reaches both.
export function sharedText(name, where) {
const file = join(sharedTextDir, `${name}.md`);
if (!existsSync(file)) throw new Error(`${where}: there's no shared/text/${name}.md`);
// Trimmed, so a trailing blank line or a CRLF ending doesn't end up in a string.
const text = readFileSync(file, "utf8").trim();
if (text.includes("{{")) {
throw new Error(`shared/text/${name}.md uses {{…}}; a snippet is a plain fact, and the text around it adds what differs per host`);
}
return text;
}

// {{name}} is a variable from vars.yaml; {{skill:name}} is that skill's name on
// the host; {{< name}} inlines shared/text/<name>.md, and {{text:name}} inserts it
// as a quoted string literal (JSON), for code.
function fill(text, values, where) {
return text.replace(/\{\{\s*([a-z_]+)(?::([a-z0-9-]+))?\s*\}\}/g, (token, name, arg) => {
text = text.replace(/\{\{<\s*([a-z0-9-]+)\s*\}\}/g, (_, name) => sharedText(name, where));
return text.replace(/\{\{\s*([a-z_][a-z0-9_]*)(?::([a-z0-9-]+))?\s*\}\}/g, (token, name, arg) => {
if (name === "skill") {
if (!values.skills.includes(arg)) {
const why = skillNames().includes(arg) ? "this host skips it (skip_skills)" : "there's no such skill";
const why = !skillNames().includes(arg)
? "there's no such skill"
: isCodeHost(values)
? "a code host (kind: code) ships no skills"
: "this host skips it (skip_skills)";
throw new Error(`${where}: ${token} names a skill this host doesn't ship: ${why}`);
}
return hostSkillName(values, arg);
}
if (name === "text" && arg !== undefined) return JSON.stringify(sharedText(arg, where));
if (arg !== undefined || !(name in values) || typeof values[name] === "object") {
throw new Error(`${where}: unknown variable ${token}`);
}
Expand Down Expand Up @@ -136,14 +187,26 @@ export function renderHost(host) {
if (!usedSlots.has(path)) throw new Error(`${path}: no skill this host ships has this slot`);
}

for (const [from, to] of Object.entries(sharedFiles)) out.set(to, readFileSync(join(root, from)));
for (const [from, to] of Object.entries(sharedDirs)) {
for (const [from, to] of isCodeHost(values) ? [] : Object.entries(sharedFiles)) out.set(to, readFileSync(join(root, from)));
for (const [from, to] of isCodeHost(values) ? [] : Object.entries(sharedDirs)) {
for (const file of listFiles(join(root, from))) out.set(`${to}/${file}`, readFileSync(join(root, from, file)));
}
for (const file of listFiles(platform)) {
if (reserved.some((r) => file === r || file.startsWith(r))) continue;
// A code host's README documents the integration here; it isn't part of the code.
if (isCodeHost(values) && file === "README.md") continue;
out.set(file, renderFile(join(platform, file), values));
}

// A token the render didn't fill (a mistyped one, or in a file type outside
// textFile) would otherwise ship as it is. Binary files (not valid UTF-8) are skipped.
for (const [path, buf] of out) {
const text = buf.toString("utf8");
const token = Buffer.from(text, "utf8").equals(buf) && text.match(leftoverToken);
if (token) {
throw new Error(`${host}: ${path} still has an unrendered ${token[0]}`);
}
}
return out;
}

Expand Down
7 changes: 5 additions & 2 deletions scripts/render.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
// render.
import { mkdirSync, readFileSync, rmSync, writeFileSync, existsSync } from "node:fs";
import { dirname, join } from "node:path";
import { distDir, hosts, listFiles, renderHost, root, vars } from "./lib.mjs";
import { distDir, hosts, isCodeHost, listFiles, renderHost, root, vars } from "./lib.mjs";

const check = process.argv.includes("--check");
const only = process.argv.slice(2).filter((a) => !a.startsWith("--"));
Expand All @@ -26,7 +26,10 @@ function hostsList() {
const rows = hosts()
.map((host) => [vars(host).host_full ?? host, host])
.sort(([a], [b]) => a.localeCompare(b))
.map(([name, host]) => `| ${name} | [dist/${host}/README.md](dist/${host}/README.md) |`);
.map(([name, host]) => {
const guide = isCodeHost(vars(host)) ? `platforms/${host}/README.md` : `dist/${host}/README.md`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s no check that platforms/<host>/README.md exists, so a code host without one leaves a dead link in the repo’s main README. I tried it and both render and validate passed.

It seems easy to run into, since CONTRIBUTING now describes a code host’s README as something that stays here rather than being shipped, so it’s a separate file to remember. Would it make sense for validate to require one when kind: code?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, validate now requires platforms/<host>/README.md for a code host (21d3148). Since that README isn't rendered, a snippet named only there no longer counts as used (c098863).

return `| ${name} | [${guide}](${guide}) |`;
});
const table = ["| Host | Setup guide |", "|---|---|", ...rows].join("\n");
const head = readme.slice(0, readme.indexOf("\n", start) + 1);
return Buffer.from(`${head}${table}\n${readme.slice(end)}`);
Expand Down
Loading
Loading