Skip to content

feat(render): code hosts and shared text snippets - #9

Merged
imoital merged 13 commits into
mainfrom
build/render-host-kinds
Nov 11, 2026
Merged

imoital merged 13 commits into
mainfrom
build/render-host-kinds

Conversation

@imoital

@imoital imoital commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

What any code host needs, so the Activepieces piece (#5) and the o11.ai listing can each be just their own folder. No host and no snippet is added here.

  • Code hosts. kind: code in vars.yaml makes a host whose output isn't a plugin but files for somewhere else: code for another project, or a listing a marketplace reads.

    • It ships only its own files, with {{…}} filled in (.ts and .js too).
    • It gets no skills, LICENSE, assets or plugin.json.
    • Its README stays out of dist/ and is linked from the hosts list.
    • It needs no release-please entry if it isn't released from here.
  • Shared text. shared/text/<name>.md holds one fact in plain text, stated once for every place that uses it:

    • a skill or text file inlines it with {{< name}};
    • code uses {{text:name}}, a quoted string literal.

    Validate fails on a snippet with markdown, more than one line, or nothing using it. The first snippets come with the hosts that use them.

The per-host scripts/ (validate.mjs, package.sh, npm run package) stay in #2, the only PR that uses them. check.sh and the AUTH…/REQS007 snippets stay in #5.

Order

Merge this before #5 and the o11.ai listing. Merging main into #5 afterwards drops these changes from its diff.

How to check it

  • npm run render -- --check passes: dist/copilot and the README are unchanged.
  • npm run validate passes.

platforms/<host>/scripts/ is now reserved like skills/: render never copies it
into dist/. npm run validate runs platforms/<host>/scripts/validate.mjs on
dist/<host> when it exists, and npm run package -- <host> runs its package.sh.
shared/text/<name>.md holds one fact in plain text. A skill inlines it with
{{< name}}; code uses {{text:name}}, a quoted string literal. using-zenrows
and extract-structured-data now take AUTH004, AUTH014, AUTH006 and REQS007's
page_not_recognised from there; every plugin renders byte-identically.
Validate fails on a snippet with markdown, more than one line, or no user.

kind: code in vars.yaml makes a host whose output isn't a plugin (code for
another project, or a marketplace listing): it ships only its own files, with
{{...}} filled in .ts/.js too, no skills, LICENSE, assets or plugin.json, keeps
its README out of dist, needs no release-please entry yet, and may keep a
scripts/check.sh.
Per-host scripts (validate.mjs, package.sh, npm run package) are only used by
the OpenAI plugin, and check.sh and the AUTH/REQS007 snippets only by the
Activepieces piece: they stay with those hosts. What's left is what any code
host needs: kind: code, and shared/text with {{< name}} and {{text:name}}.
Comments and CONTRIBUTING describe any code host, a piece or a listing.
@imoital imoital changed the title feat(render): host tooling, shared text snippets and code hosts feat(render): code hosts and shared text snippets Nov 10, 2026
@imoital
imoital requested a review from tedi-z November 10, 2026 16:19
Comment thread scripts/validate.mjs Outdated
const name = file.replace(/\.md$/, "");
const text = readFileSync(join(sharedTextDir, file), "utf8");
if (!used.has(name)) errors.push(`shared/text/${file} isn't used by any skill or host`);
if (/[`*_[\]#]/.test(text)) errors.push(`shared/text/${file} has markdown; keep snippets plain text so code can use them`);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this check is going to block most of the snippets you’d actually want to write. It rejects _, and our parameter names are all snake_case.

Here’s the first snippet I tried, which feels like the canonical example:

shared/text/cost.md: "A request with js_render costs 5 credits."
→ ❌ shared/text/cost.md has markdown; keep snippets plain text so code can use them

I grepped the shared skills to check whether the underscore ban is earning anything. js_render appears 11 times, css_extractor and proxy_country 10 each, premium_proxy 9, and there isn’t a single place where anyone uses underscores for emphasis. So we’re banning a character nobody writes as markdown, and it happens to be the character that shows up in every parameter name. Same story for [, ] and #: those only mean something in a link or a heading, not in the middle of a sentence like “returns [{...}]”.

Backticks and * are worth keeping, since those genuinely look wrong inside a TS string. Could we narrow it to just those plus link and heading syntax?

if (/[`*]/.test(text) || /\]\(/.test(text) || /^#/.test(text.trim()))

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, narrowed it to your rule: backticks, *, ]( and a leading #. js_render and [{...}] now pass. Fixed in 21d3148, and npm test has a case for each.

Comment thread scripts/validate.mjs Outdated
const used = new Set();
for (const file of snippetSources) {
if (!textFile.test(file)) continue;
for (const [, , name] of readFileSync(file, "utf8").matchAll(/\{\{\s*(<\s*|text:)([a-z0-9-]+)\s*\}\}/g)) used.add(name);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s a one-character gap between this regex and the one in lib.mjs:119, and an unrendered placeholder can slip through it.

This one allows a space between {{ and <. The one in lib.mjs doesn’t. So if someone writes {{ < cost}} instead of {{< cost}}, this check counts it as a use of the snippet (which keeps the unused-snippet rule on line 242 happy), but the render never substitutes it.

I tried it on this branch and {{ < cost}} ended up verbatim in dist/copilot/README.md, with both npm run render and npm run validate passing. The guard I suggested over on lib.mjs:22 would catch this one too, if you’d rather fix both in one go.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Validate now matches {{< the way the render does, and the render guard (next thread) fails on the spaced form, so it can't ship. Fixed in 21d3148, with a test case.

Comment thread scripts/lib.mjs
const reserved = ["vars.yaml", "overrides.lock", "skills/"];
// {{vars}} are filled in files with these extensions; others are copied as bytes.
const textFile = /\.(json|md|mdc|ya?ml|txt|toml)$/;
export const textFile = /\.(json|md|mdc|ya?ml|txt|toml|[cm]?[jt]s)$/;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allowlist fails quietly, which was fine when the output was manifests and markdown but is riskier now that a code host’s whole output is code. Anything with an extension that isn’t on the list gets copied byte for byte, placeholders and all, with no warning.

I set up a kind: code host to see the effect:

platforms/piece/comp.tsx → dist/piece/comp.tsx:  export const a = "{{host_full}}";
platforms/piece/setup.py → dist/piece/setup.py:  name = "{{host_full}}"

Both render and validate were clean. .tsx, .jsx, .vue, .py and .go all land here.

One option instead of extending the regex each time a host brings a new file type is to check the rendered output. Something like this at the end of renderHost, once out is built:

for (const [path, buf] of out) {
  const text = buf.toString("utf8");
  if (!Buffer.compare(Buffer.from(text, "utf8"), buf) && text.includes("{{")) {
    throw new Error(`${host}: ${path} still has an unrendered {{…}}`);
  }
}

It matches what CONTRIBUTING already describes (“Unknown variables … fail the render”), and it covers the {{ < name}} case above and any future variation of the same thing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took the idea of checking the output: renderHost now fails on any of this tool's tokens left in a rendered file ({{name}}, {{< x}}, {{text:x}}, …), whatever its extension, and the error names the leftover. I first made it fail on any {{, but that blocked JSX style={{…}} and Make/n8n expressions, which code hosts will need, so it only matches our own token shapes (9cb7a04). The allow-list still decides which files get filled. In #5 I'll reword the one doc comment that mentions {{text:…}} when it merges main.

Comment thread scripts/render.mjs
.sort(([a], [b]) => a.localeCompare(b))
.map(([name, host]) => `| ${name} | [dist/${host}/README.md](dist/${host}/README.md) |`);
.map(([name, host]) => {
const guide = isCodeHost(vars(host)) ? `platforms/${host}/README.md` : `dist/${host}/README.md`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s no check that platforms/<host>/README.md exists, so a code host without one leaves a dead link in the repo’s main README. I tried it and both render and validate passed.

It seems easy to run into, since CONTRIBUTING now describes a code host’s README as something that stays here rather than being shipped, so it’s a separate file to remember. Would it make sense for validate to require one when kind: code?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, validate now requires platforms/<host>/README.md for a code host (21d3148). Since that README isn't rendered, a snippet named only there no longer counts as used (c098863).

Comment thread scripts/lib.mjs Outdated
// skill of that name). Returns [{ name, own }].
export function hostSkills(host, values = vars(host)) {
// A code host (kind: code) ships no skills.
if (isCodeHost(values)) return [];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning [] means a code host’s skills/ folder is skipped silently. I added a platforms/piece/skills/scrape/SKILL.md and it wasn’t shipped and nothing was reported. Validate also reported piece: 0 whole-file overrides, 5 skill lines of its own for a host that ships no skills, which makes the report line a little misleading.

The comparable case on line 163 (an unused slot file) throws, so throwing here when a code host has anything under skills/ would be consistent.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now it throws, like an unused slot. The report line for a code host reads "code host, ships only its own files", and {{skill:x}} in a code host says "a code host ships no skills". Fixed in 21d3148.

Fail the render on any {{…}} left in a rendered text file, and on a skills/ folder in a code host. Validate requires a code host README without {{…}}, matches {{< name}} the way the render does, and only treats backticks, *, links and headings as markdown in a snippet.
@imoital
imoital requested a review from a team as a code owner November 10, 2026 16:55
@imoital
imoital requested a review from tedi-z November 11, 2026 08:36
@imoital

imoital commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for trying to break it. Your cases, plus what a second adversarial pass found (trimmed snippets, no {{…}} inside a snippet, a clearer error for a bad kind: or vars.yaml), are now in npm test (scripts/test.mjs, 21 cases, run by CI). I also trial-merged all the open PRs on top of this in merge order: everything renders, validates and passes the tests. Ready for another look.

tedi-z
tedi-z previously approved these changes Nov 11, 2026
Comment thread scripts/lib.mjs Outdated
// fill, or with a stray space or capital letter the fill regexes don't take.
// Other {{…}} (n8n or Make expressions, JSX's style={{…}}) isn't ours and ships
// as written.
export const leftoverToken = /\{\{\s*(?:[<>]\??\s*[a-z0-9_-]+|[a-z_]+(?:\s*:\s*[a-z0-9-]+)?)\s*\}\}/i;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not blocking. The comment above says JSX’s style={{…}} ships as written, but the [a-z_]+(?:\s*:\s*[a-z0-9-]+)? branch catches it whenever the value isn’t quoted:

style={{ color: accent }}  → ❌ still has an unrendered {{ color: accent }}
style={{ width: 100 }}     → ❌ still has an unrendered {{ width: 100 }}
style={{ color: "red" }}   → passes

The case in test.mjs passes because of the quotes. Since the only colon forms we emit are skill: and text:, narrowing to (?:skill|text)\s*:\s*[a-z0-9-]+ would cover it. style={{ color }} would still trip the bare-word branch, but that’s what catches {{host_full}}, so probably unavoidable.

Separately: is there a way to ship a literal {{word}}? "{{trigger}}" in a code host’s .json fails as an unknown variable and I couldn’t find an escape. Dotted and $ forms survive, so n8n is fine either way.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch: my test used style={{ color: "red" }}, the one JSX form the widened pattern didn't match, so it stayed green. Took your rule as is: a colon marks one of our tokens only after skill or text, so {{ flex: 1 }} and {{flex:1}} ship as written. A mistyped keyword like {{txet:cost}} now looks like another tool's {{a:b}} and ships too; that's noted in the test file as the known limit.

A second pass over the pattern found two more gaps, fixed in the same commit (0fe5354): _ or . in an argument ({{skill:scrape_webpage}}, {{< cost.md}}) wasn't caught, and a variable with a digit ({{api_v2}}) was neither filled nor caught. The fill now takes digits in a name, and the guard takes _, ., digits and hyphens where the fill doesn't.

The test is now a table: 21 forms of our tokens that must be caught, and JSX with a string, a number, a variable and no spaces, Make, Zapier, n8n and Handlebars that must ship untouched.

On a literal {{word}}: there's no escape today, on purpose. The platforms we might ship to write their placeholders differently (Make and Zapier as {{a.b}}, n8n as {{ $json.x }}), and the render leaves those alone. So nothing needs the bare form yet; the first host that does gets an escape, in that host's PR.

…digits

A colon marks one of our tokens only after skill or text, so JSX's style={{ flex: 1 }} ships as written. Arguments may contain _ and ., names may contain digits and hyphens, and a variable such as api_v2 is now filled instead of skipped.
@tedi-z

tedi-z commented Nov 11, 2026

Copy link
Copy Markdown

LGTM! :)

@imoital
imoital merged commit 3768e17 into main Nov 11, 2026
3 checks passed
@imoital
imoital deleted the build/render-host-kinds branch November 11, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants