Repository navigation
feat(render): code hosts and shared text snippets - #9
Conversation
platforms/<host>/scripts/ is now reserved like skills/: render never copies it into dist/. npm run validate runs platforms/<host>/scripts/validate.mjs on dist/<host> when it exists, and npm run package -- <host> runs its package.sh.
shared/text/<name>.md holds one fact in plain text. A skill inlines it with
{{< name}}; code uses {{text:name}}, a quoted string literal. using-zenrows
and extract-structured-data now take AUTH004, AUTH014, AUTH006 and REQS007's
page_not_recognised from there; every plugin renders byte-identically.
Validate fails on a snippet with markdown, more than one line, or no user.
kind: code in vars.yaml makes a host whose output isn't a plugin (code for
another project, or a marketplace listing): it ships only its own files, with
{{...}} filled in .ts/.js too, no skills, LICENSE, assets or plugin.json, keeps
its README out of dist, needs no release-please entry yet, and may keep a
scripts/check.sh.
Per-host scripts (validate.mjs, package.sh, npm run package) are only used by
the OpenAI plugin, and check.sh and the AUTH/REQS007 snippets only by the
Activepieces piece: they stay with those hosts. What's left is what any code
host needs: kind: code, and shared/text with {{< name}} and {{text:name}}.
Comments and CONTRIBUTING describe any code host, a piece or a listing.
| const name = file.replace(/\.md$/, ""); | ||
| const text = readFileSync(join(sharedTextDir, file), "utf8"); | ||
| if (!used.has(name)) errors.push(`shared/text/${file} isn't used by any skill or host`); | ||
| if (/[`*_[\]#]/.test(text)) errors.push(`shared/text/${file} has markdown; keep snippets plain text so code can use them`); |
There was a problem hiding this comment.
I think this check is going to block most of the snippets you’d actually want to write. It rejects _, and our parameter names are all snake_case.
Here’s the first snippet I tried, which feels like the canonical example:
shared/text/cost.md: "A request with js_render costs 5 credits."
→ ❌ shared/text/cost.md has markdown; keep snippets plain text so code can use them
I grepped the shared skills to check whether the underscore ban is earning anything. js_render appears 11 times, css_extractor and proxy_country 10 each, premium_proxy 9, and there isn’t a single place where anyone uses underscores for emphasis. So we’re banning a character nobody writes as markdown, and it happens to be the character that shows up in every parameter name. Same story for [, ] and #: those only mean something in a link or a heading, not in the middle of a sentence like “returns [{...}]”.
Backticks and * are worth keeping, since those genuinely look wrong inside a TS string. Could we narrow it to just those plus link and heading syntax?
if (/[`*]/.test(text) || /\]\(/.test(text) || /^#/.test(text.trim()))
There was a problem hiding this comment.
Agreed, narrowed it to your rule: backticks, *, ]( and a leading #. js_render and [{...}] now pass. Fixed in 21d3148, and npm test has a case for each.
| const used = new Set(); | ||
| for (const file of snippetSources) { | ||
| if (!textFile.test(file)) continue; | ||
| for (const [, , name] of readFileSync(file, "utf8").matchAll(/\{\{\s*(<\s*|text:)([a-z0-9-]+)\s*\}\}/g)) used.add(name); |
There was a problem hiding this comment.
There’s a one-character gap between this regex and the one in lib.mjs:119, and an unrendered placeholder can slip through it.
This one allows a space between {{ and <. The one in lib.mjs doesn’t. So if someone writes {{ < cost}} instead of {{< cost}}, this check counts it as a use of the snippet (which keeps the unused-snippet rule on line 242 happy), but the render never substitutes it.
I tried it on this branch and {{ < cost}} ended up verbatim in dist/copilot/README.md, with both npm run render and npm run validate passing. The guard I suggested over on lib.mjs:22 would catch this one too, if you’d rather fix both in one go.
There was a problem hiding this comment.
Good catch. Validate now matches {{< the way the render does, and the render guard (next thread) fails on the spaced form, so it can't ship. Fixed in 21d3148, with a test case.
| const reserved = ["vars.yaml", "overrides.lock", "skills/"]; | ||
| // {{vars}} are filled in files with these extensions; others are copied as bytes. | ||
| const textFile = /\.(json|md|mdc|ya?ml|txt|toml)$/; | ||
| export const textFile = /\.(json|md|mdc|ya?ml|txt|toml|[cm]?[jt]s)$/; |
There was a problem hiding this comment.
This allowlist fails quietly, which was fine when the output was manifests and markdown but is riskier now that a code host’s whole output is code. Anything with an extension that isn’t on the list gets copied byte for byte, placeholders and all, with no warning.
I set up a kind: code host to see the effect:
platforms/piece/comp.tsx → dist/piece/comp.tsx: export const a = "{{host_full}}";
platforms/piece/setup.py → dist/piece/setup.py: name = "{{host_full}}"
Both render and validate were clean. .tsx, .jsx, .vue, .py and .go all land here.
One option instead of extending the regex each time a host brings a new file type is to check the rendered output. Something like this at the end of renderHost, once out is built:
for (const [path, buf] of out) {
const text = buf.toString("utf8");
if (!Buffer.compare(Buffer.from(text, "utf8"), buf) && text.includes("{{")) {
throw new Error(`${host}: ${path} still has an unrendered {{…}}`);
}
}
It matches what CONTRIBUTING already describes (“Unknown variables … fail the render”), and it covers the {{ < name}} case above and any future variation of the same thing.
There was a problem hiding this comment.
Took the idea of checking the output: renderHost now fails on any of this tool's tokens left in a rendered file ({{name}}, {{< x}}, {{text:x}}, …), whatever its extension, and the error names the leftover. I first made it fail on any {{, but that blocked JSX style={{…}} and Make/n8n expressions, which code hosts will need, so it only matches our own token shapes (9cb7a04). The allow-list still decides which files get filled. In #5 I'll reword the one doc comment that mentions {{text:…}} when it merges main.
| .sort(([a], [b]) => a.localeCompare(b)) | ||
| .map(([name, host]) => `| ${name} | [dist/${host}/README.md](dist/${host}/README.md) |`); | ||
| .map(([name, host]) => { | ||
| const guide = isCodeHost(vars(host)) ? `platforms/${host}/README.md` : `dist/${host}/README.md`; |
There was a problem hiding this comment.
There’s no check that platforms/<host>/README.md exists, so a code host without one leaves a dead link in the repo’s main README. I tried it and both render and validate passed.
It seems easy to run into, since CONTRIBUTING now describes a code host’s README as something that stays here rather than being shipped, so it’s a separate file to remember. Would it make sense for validate to require one when kind: code?
| // skill of that name). Returns [{ name, own }]. | ||
| export function hostSkills(host, values = vars(host)) { | ||
| // A code host (kind: code) ships no skills. | ||
| if (isCodeHost(values)) return []; |
There was a problem hiding this comment.
Returning [] means a code host’s skills/ folder is skipped silently. I added a platforms/piece/skills/scrape/SKILL.md and it wasn’t shipped and nothing was reported. Validate also reported piece: 0 whole-file overrides, 5 skill lines of its own for a host that ships no skills, which makes the report line a little misleading.
The comparable case on line 163 (an unused slot file) throws, so throwing here when a code host has anything under skills/ would be consistent.
There was a problem hiding this comment.
Now it throws, like an unused slot. The report line for a code host reads "code host, ships only its own files", and {{skill:x}} in a code host says "a code host ships no skills". Fixed in 21d3148.
Fail the render on any {{…}} left in a rendered text file, and on a skills/ folder in a code host. Validate requires a code host README without {{…}}, matches {{< name}} the way the render does, and only treats backticks, *, links and headings as markdown in a snippet.
… with every tracked file
…s and vars.yaml
Other tools' {{…}} (JSX, Make, n8n) ships as written. Snippets are trimmed and can't contain {{…}}, and a malformed vars.yaml or kind names the file.
… or vars.yaml as used
|
Thanks for trying to break it. Your cases, plus what a second adversarial pass found (trimmed snippets, no |
| // fill, or with a stray space or capital letter the fill regexes don't take. | ||
| // Other {{…}} (n8n or Make expressions, JSX's style={{…}}) isn't ours and ships | ||
| // as written. | ||
| export const leftoverToken = /\{\{\s*(?:[<>]\??\s*[a-z0-9_-]+|[a-z_]+(?:\s*:\s*[a-z0-9-]+)?)\s*\}\}/i; |
There was a problem hiding this comment.
Not blocking. The comment above says JSX’s style={{…}} ships as written, but the [a-z_]+(?:\s*:\s*[a-z0-9-]+)? branch catches it whenever the value isn’t quoted:
style={{ color: accent }} → ❌ still has an unrendered {{ color: accent }}
style={{ width: 100 }} → ❌ still has an unrendered {{ width: 100 }}
style={{ color: "red" }} → passes
The case in test.mjs passes because of the quotes. Since the only colon forms we emit are skill: and text:, narrowing to (?:skill|text)\s*:\s*[a-z0-9-]+ would cover it. style={{ color }} would still trip the bare-word branch, but that’s what catches {{host_full}}, so probably unavoidable.
Separately: is there a way to ship a literal {{word}}? "{{trigger}}" in a code host’s .json fails as an unknown variable and I couldn’t find an escape. Dotted and $ forms survive, so n8n is fine either way.
There was a problem hiding this comment.
Good catch: my test used style={{ color: "red" }}, the one JSX form the widened pattern didn't match, so it stayed green. Took your rule as is: a colon marks one of our tokens only after skill or text, so {{ flex: 1 }} and {{flex:1}} ship as written. A mistyped keyword like {{txet:cost}} now looks like another tool's {{a:b}} and ships too; that's noted in the test file as the known limit.
A second pass over the pattern found two more gaps, fixed in the same commit (0fe5354): _ or . in an argument ({{skill:scrape_webpage}}, {{< cost.md}}) wasn't caught, and a variable with a digit ({{api_v2}}) was neither filled nor caught. The fill now takes digits in a name, and the guard takes _, ., digits and hyphens where the fill doesn't.
The test is now a table: 21 forms of our tokens that must be caught, and JSX with a string, a number, a variable and no spaces, Make, Zapier, n8n and Handlebars that must ship untouched.
On a literal {{word}}: there's no escape today, on purpose. The platforms we might ship to write their placeholders differently (Make and Zapier as {{a.b}}, n8n as {{ $json.x }}), and the render leaves those alone. So nothing needs the bare form yet; the first host that does gets an escape, in that host's PR.
…digits
A colon marks one of our tokens only after skill or text, so JSX's style={{ flex: 1 }} ships as written. Arguments may contain _ and ., names may contain digits and hyphens, and a variable such as api_v2 is now filled instead of skipped.
|
LGTM! :) |
Summary
What any code host needs, so the Activepieces piece (#5) and the o11.ai listing can each be just their own folder. No host and no snippet is added here.
Code hosts.
kind: codeinvars.yamlmakes a host whose output isn't a plugin but files for somewhere else: code for another project, or a listing a marketplace reads.{{…}}filled in (.tsand.jstoo).LICENSE, assets orplugin.json.dist/and is linked from the hosts list.Shared text.
shared/text/<name>.mdholds one fact in plain text, stated once for every place that uses it:{{< name}};{{text:name}}, a quoted string literal.Validate fails on a snippet with markdown, more than one line, or nothing using it. The first snippets come with the hosts that use them.
The per-host
scripts/(validate.mjs,package.sh,npm run package) stay in #2, the only PR that uses them.check.shand theAUTH…/REQS007snippets stay in #5.Order
Merge this before #5 and the o11.ai listing. Merging
maininto #5 afterwards drops these changes from its diff.How to check it
npm run render -- --checkpasses:dist/copilotand the README are unchanged.npm run validatepasses.