Skip to content

fix(api): reject webhook body templates that don't compile, add /api/v2/webhooks/validate - #3365

Merged
kodiakhq[bot] merged 5 commits into
mainfrom
brandon/webhook-backend-validation
Oct 14, 2026
Merged

kodiakhq[bot] merged 5 commits into
mainfrom
brandon/webhook-backend-validation

Conversation

@brandon-pereira

Copy link
Copy Markdown
Member

Builds on #3350 (base is that branch; retarget to main once it merges).

Why

#3350 shows Handlebars compile errors in the webhook form, but saving still goes through, and the External API v2 and MCP never checked the body. A body that doesn't compile fails on every alert delivery. This rejects it at write time. It also adds the /validate endpoint requested on #3350 so Terraform can check webhook configs at plan time.

What changed

  • Generic and incident.io body templates that don't compile now get a 400 on create, update and test, with the Handlebars message plus line/column, e.g. Invalid webhook body template (line 1, column 29): Parse error on line 1: …. Slack and empty bodies are skipped.
    • Internal API: POST /webhooks, PUT /webhooks/:id, POST /webhooks/test (rejects before sending).
    • External API v2 and MCP clickstack_save_webhook get the check through createWebhook/updateWebhook. MCP returns it as a user error.
  • New POST /api/v2/webhooks/validate, modelled on POST /api/v2/dashboards/validate. It parses with externalWebhookCreateSchema, runs the same URL and body checks as create, never saves, and always returns 200 with { valid, errors: [{ path, message }], normalized }. It doesn't check name uniqueness. openapi.json is regenerated.
  • The check only compiles the template. It deliberately doesn't render it and parse the output as JSON.
  • The MCP body description and docs/alert-webhook-template-variables.md now say these templates are rejected.

Tests

make dev-int FILE=webhooks: 154 passed. Coverage:

  • 400 on create and update for the internal API, External API v2 and MCP
  • /test rejects without sending
  • {{#if (eq state "ALERT")}}… still saves
  • /validate: valid, invalid body, URL and body errors together, schema errors, 401 without a key

@changeset-bot

changeset-bot Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 9bc49f9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 14, 2026 10:31pm UTC
hyperdx-storybook Ready Ready Preview Oct 14, 2026 10:31pm UTC

Request Review

Comment thread packages/api/src/utils/validators.ts Resolved
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 3 minor

2 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/routers/external-api/v2/webhooks.ts:737 — Docs and the OpenAPI description leave out the partial and decorator rejections → The validator also rejects undefined partials ({{> foo}}) and any decorator other than inline. The body description here, the regenerated openapi.json, the MCP body description in saveWebhook.ts, and docs/alert-webhook-template-variables.md only mention compile errors and helpers other than eq/built-ins. Add 'or uses a partial it does not define inline, or a decorator other than inline' to each.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Critical impact] Adds validation to webhook body templates and new API endpoint.

Fix the rejection of working dynamic partials before merging.

Fix All in Claude CodeFindings

  1. P1 Working dynamic partials are rejected ▶
  2. P2 Skipped definitions pass validation ▶

Summary

The PR checks webhook body templates before saving or testing them and adds a read-only /api/v2/webhooks/validate endpoint.

  • Webhook writes and test sends reject templates that delivery cannot use.
  • External API v2 checks webhook configs without saving them.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Write[Create or update through API or MCP] --> Checks[Check URL and body template]
  Test[Send test webhook] --> Checks
  Checks -->|Invalid| Error[Return client error]
  Checks -->|Valid write| Save[Save webhook]
  Checks -->|Valid test| Send[Send test request]
  Validate[POST /api/v2/webhooks/validate] --> Schema[Parse create schema]
  Schema --> Collect[Collect URL and body errors]
  Collect --> Result[Return result without saving]
Loading

Reviews (6) · Last reviewed commit: "fix(api): allow @partial-block, move bod..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The validation logic is well-constructed: the custom AST walk in findUndeliverableConstruct faithfully mirrors the delivery renderer's helperMissing/blockHelperMissing semantics (bare {{x}} lookups and {{#x}} block-context values pass; only helper calls with arguments are rejected), partial blocks and @partial-block are correctly allowed, and /validate shares the single WEBHOOK_CONFIG_CHECKS list with create/update so the two paths cannot drift. Auth is correct — /validate sits behind defaultRateLimiter + validateUserAccessKey and only echoes the caller's own input with headers/queryParams stripped. All three substantive prior-review threads (@partial-block rejection, the file-size split into webhookBodyTemplate.ts, and sharing checks between /validate and create) are addressed.

🟡 P2 -- recommended

  • packages/api/src/routers/external-api/v2/webhooks.ts:624 -- Create, update, and test on the External API v2 now return 400 for a webhook body that previously saved successfully, which is a behavior change to a public, Terraform-facing API surface shipped under a patch changeset.
    • Fix: Confirm the rejection of previously-accepted non-compiling bodies is called out as a behavior change in the release notes so existing clients re-applying a stored broken body understand the new 400.
  • packages/api/src/routers/external-api/__tests__/webhooks.int.test.ts:996 -- The /validate suite covers the 401 unauthenticated path but not the handler's 403 authenticated-but-no-team branch, leaving a new response branch unexercised.
    • Fix: Add a /validate test that authenticates a user without a team and asserts the 403 response.
🔵 P3 nitpicks (1)
  • packages/api/src/utils/webhookBodyTemplate.ts:81 -- The Decorator visitor flags every non-block decorator as Unknown decorator, including inline ({{*inline "p"}}), which the delivery renderer's Handlebars.create() actually registers, so a template using the rarely-used non-block inline form is rejected despite rendering.
    • Fix: Allow a non-block inline decorator in the Decorator handler, mirroring the DecoratorBlock case, or document that only the block form is supported.

Reviewers (2): ce-security-reviewer (returned clean), orchestrator correctness/adversarial/previous-comments analysis.

Testing gaps:

  • No test pins the /validate normalized output to exactly the readable key set, so a future schema addition that introduces a new write-only field could silently leak into normalized.
  • No test exercises a near-cap (16384-char) or deeply nested body against the parser/AST-walk cost bound.
  • Coverage note: of the 9 dispatched persona reviewers, only ce-security-reviewer had returned at synthesis time; correctness, adversarial, testing, maintainability, api-contract, typescript, project-standards, and previous-comments findings reflect the orchestrator's own direct analysis of the diff and surrounding code rather than their independent sub-agent runs.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

Comment thread packages/api/src/utils/validators.ts Resolved
Comment thread packages/api/src/utils/validators.ts Resolved
Comment thread packages/api/src/utils/validators.ts Resolved
Comment thread packages/api/src/utils/validators.ts Resolved
Comment thread packages/api/src/utils/validators.ts Resolved
Comment thread packages/api/src/utils/validators.ts Resolved
const partialRefs: hbs.AST.PartialStatement[] = [];

const checkHelperCall = (node: HelperCall) => {
if (found != null || node.params.length === 0) return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Built-in block helpers called with no argument pass validation but throw on every delivery

checkHelperCall returns early when node.params.length === 0, so {{#if}}x{{/if}}, {{#unless}}…, {{#with}}… and {{#each}}… are accepted. Stock Handlebars throws #if requires exactly one argument / Must pass iterator to #each when these render, and delivery uses stock Handlebars (createHandlebarsWithHelpers in tasks/checkAlerts/transports/generic.ts). That breaks the 'throws on every delivery' guarantee in the doc comment. Fix: reject if/unless/with/each BlockStatements where params.length !== 1.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

);
}
super.DecoratorBlock(node);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Inline partials are treated as global, so a partial used outside the block that defines it passes

inlinePartials is one flat set for the whole template. But {{#*inline}} only registers its partial inside the program where it appears. So {{#if title}}{{#*inline "p"}}x{{/inline}}{{/if}}{{> p}} passes validation and then fails on every render with 'The partial p could not be found'. Fix: track inline definitions per Program scope, using a stack pushed and popped in Program, and resolve each PartialStatement against the scopes that enclose it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

service,
body,
}: {
service: WebhookService;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Unknown-helper/partial check lives only in the API, so the webhook form's inline error never shows it

packages/app/src/components/TeamSettings/WebhookForm.tsx:148 shows inline errors by calling getWebhookTemplateError from common-utils core/handlebarsEnv.ts, and that only catches compile errors. A body like {{#if (equals state "ALERT")}} shows no inline error. Save then fails with only a generic toast built from the 400 message. Fix: move findUndeliverableConstruct and the helper list into common-utils next to getWebhookTemplateError (or have that function call it) so the form and the API use one check. Have createHandlebarsWithHelpers register from that shared list too; that also removes the hand-synced WEBHOOK_BODY_HELPERS copy and the import-cycle workaround.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Comment on lines +113 to +114
// `@`-prefixed partials, like `@partial-block`, come from Handlebars.
return !name.startsWith('@') && !inlinePartials.has(name);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unknown partials pass validation

The new startsWith('@') exemption accepts unknown partials such as {{> @missing}}. Handlebars does not supply arbitrary @ partials, and delivery registers none. This body now passes create and /validate but fails when an alert renders it. Limit the exemption to @partial-block where Handlebars supplies it.

Knowledge Base Used: Alerting and notifications

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Nice one

@brandon-pereira
brandon-pereira changed the base branch from brandon/brandon-json-parsing-issue to main October 14, 2026 22:23
@brandon-pereira
brandon-pereira dismissed jordan-simonovski’s stale review October 14, 2026 22:23

The base branch was changed.

Validate generic and incident.io bodies with getWebhookTemplateError on
create, update, and test across the internal API, External API v2, and
MCP. Add POST /api/v2/webhooks/validate for plan-time checks.
@brandon-pereira
brandon-pereira force-pushed the brandon/webhook-backend-validation branch from c466891 to 9bc49f9 Compare October 14, 2026 22:27
@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Oct 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Security-critical files (1) — auth or input validation, escalated at any size:
    • packages/api/src/utils/validators.ts
  • Critical-path files (1) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/webhooks.ts

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 7
  • Production lines changed: 492 (+ 484 in test files, excluded from tier calculation)
  • Critical-path lines changed: 183
  • Branch: brandon/webhook-backend-validation
  • Author: brandon-pereira

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/api/src/routers/external-api/v2/dashboards.ts:2748-2760 ↔ packages/api/src/routers/external-api/v2/webhooks.ts:534-546 — 13 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 125 in total (1.71%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

checkHelperCall(node);
super.SubExpression(node);
}
PartialStatement(node: hbs.AST.PartialStatement) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Inline partials are collected template-wide, but Handlebars scopes them to the block that defines them

inlinePartials is one flat set, so {{#if x}}{{#*inline "p"}}y{{/inline}}{{/if}}{{> p}} passes validation. At delivery it throws The partial p could not be found, because an inline partial is only visible inside the program that declares it and that program's children. This is the kind of always-failing body the PR says it rejects. Track inline definitions per Program scope: push a scope on entering a Program and pop it on exit, then resolve each PartialStatement against the scopes enclosing it. Add a test with the definition nested inside a block.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

});

const nameOf = (node: hbs.AST.Expression): string =>
'original' in node ? String(node.original) : node.type;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The unknown-helper/partial/decorator check is API-only, so the webhook form still lets these bodies through to a 400 on save

getWebhookTemplateError lives in @hyperdx/common-utils/dist/core/handlebarsEnv and WebhookForm.tsx:148 runs it inline. findUndeliverableConstruct is only in packages/api, so the form shows no error for {{#if (equals state "ALERT")}} and the user only finds out when save returns a 400. Move findUndeliverableConstruct and WEBHOOK_BODY_HELPERS into handlebarsEnv.ts next to getWebhookTemplateError, or fold them into it. That gives the API and the form one check. It also removes the import-cycle reason given for keeping the helper list here.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Comment on lines +111 to +114
const unknownPartial = partialRefs.find(ref => {
const name = nameOf(ref.name);
// `@`-prefixed partials, like `@partial-block`, come from Handlebars.
return !name.startsWith('@') && !inlinePartials.has(name);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Working dynamic partials are rejected

The partial check reports Unknown partial "SubExpression" for every dynamic name, even when it resolves to a defined inline partial. For example, {{#*inline "p"}}{"text":"hello"}{{/inline}}{{> (lookup "p" 0)}} renders a valid body with the helpers delivery provides, but create, update, test, and /validate now reject it.

Do not treat a dynamic name as a missing literal partial.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

Comment on lines +94 to +97
DecoratorBlock(node: hbs.AST.DecoratorBlock) {
const [partial] = node.params;
if (nameOf(node.path) === 'inline' && partial?.type === 'StringLiteral') {
inlinePartials.add(nameOf(partial));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Skipped definitions pass validation

The inline partial check collects definitions from every branch into one set. A body such as {{#if false}}{{#*inline "p"}}{"text":"hello"}{{/inline}}{{/if}}{{> p}} therefore passes validation, although the branch that defines p never runs. Delivery has no registered p, so rendering fails before any webhook request is sent.

Check whether each definition is available where the partial is used.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

@github-actions

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 417 passed • 1 skipped • 1285s

Status Count
✅ Passed 417
❌ Failed 0
⚠️ Flaky 0
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@kodiakhq
kodiakhq Bot merged commit ecddea2 into main Oct 14, 2026
33 checks passed
@kodiakhq
kodiakhq Bot deleted the brandon/webhook-backend-validation branch October 14, 2026 22:55

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 9bc49f90 Deployed Oct 14, 2026 by vercel[bot]
Preview – hyperdx-storybook — 9bc49f90 Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants