Import REST tools from a HAR file (browser network capture)

[rangarajan19]

Closes #1005.

Summary

• New HarParser (packages/backend/src/connectors/parsers/har.parser.ts): validates HAR 1.2, keeps only entries under the connector's base URL, skips OPTIONS/HEAD, static assets and WebSocket entries, collapses same-template requests (/orders/123 + /orders/456 → one tool with {id}), and maps query strings and JSON/form bodies to tool parameters.
• Credential scrubbing: drops Cookie, Authorization, Proxy-Authorization, HTTP/2 pseudo-headers and any header/query/body field named like a credential (token, secret, password, api_key, session, csrf, xsrf, auth, signature, sig). A credential-named body field (e.g. a login form's password) still becomes a parameter, just with no captured default.
• 'har' added as an ImportToolsDto.source, wired through the existing additive-import path (like curl/json); the response now reports how many entries were skipped and why (other-origin, static/socket, duplicate).
• Frontend: new "HAR file (browser capture)" import source with a file picker. The file is slimmed in the browser before upload — response.content, request/response cookies, and credential-looking headers (including Set-Cookie) are stripped — with a size check so an oversized capture after slimming is rejected in the panel instead of failing the request with a 413 (per review feedback on the issue).
• Docs: new "From a HAR file" section in docs/connectors/rest.md, example added to docs/api-reference.md.

Why

Many internal tools and SaaS products have an API but no OpenAPI spec or Postman collection — the only description of it is what their own web app sends. A HAR file (DevTools → Network → "Save all as HAR") records exactly those requests.

Test plan

• cd packages/backend && npx jest src/connectors/parsers/har.parser.spec.ts — 8/8 passing (origin/static-asset filtering, path-id collapsing, query/body mapping, base-URL-prefix handling, credential-scrubbing assertion over the full serialized output, determinism, invalid-file rejection).
• cd packages/backend && npx jest src/connectors — 1343/1343 passing (no regressions from the new constructor parameter on ConnectorsController).
• npx tsc --noEmit clean on both packages/backend and packages/frontend.
• npm run lint clean on both packages (no new warnings).
• Manual, end-to-end: captured a real HAR from jsonplaceholder.typicode.com (GET /posts/1, GET /posts/2, GET /comments?postId=1, POST /posts), created a REST connector against that base URL on a local instance, imported the HAR through the new UI — got "created 3, updated 0 (skipped 0 other-origin, 0 static/socket, 1 duplicate entries)" — and ran the generated get_posts tool with id: 1, which returned the real API response.

Out of scope (per the issue)

Response-schema/outputSchema inference, auto-detecting or configuring the connector's auth scheme, creating a new connector from a HAR in one step, and splitting GraphQL requests in a HAR into separate tools.

[helpcodeai]

Thanks @rangarajan19, this is a solid start: parser, tests and docs are all there and CI is green. Before merging, the import has to make sure no captured credential survives. #1005 says "never keeps a credential value", and a HAR file is full of them. I ran the parser locally on a crafted HAR, and these cases got through:

Must fix

1. Nested body secrets are kept as defaults (har.parser.ts ~215–217). A top-level body key that isn't credential-named keeps its whole value as the default. {"user":{"password":"hunter2…"}} became a user parameter whose default contains the password, and every org member and the AI client can see it. Either keep only scalar defaults, or walk the body recursively and drop credential-looking keys and values.

2. Credentials whose names miss the regex survive:

   • ?key=AIza…
   • Ocp-Apim-Subscription-Key
   • X-Access-Key
   • X-Firebase-AppCheck (a JWT)
   • X-Custom: Bearer …

   Please reuse the shared helpers in packages/backend/src/common/secret-literals.util.ts (isCredentialHeaderName, isSecretQueryName, findSecretLiterals / isCredentialLiteral) on top of the issue's regex, and check values too (Bearer , JWTs, known key prefixes). Since #1054, imports also go through connectors/secret-extraction.util.ts. A HAR import should drop captured values rather than turn them into connector env vars: they belong to the user's browser session, not to the connector.

3. Header allowlist. Every other header is frozen as a static tool header today, including If-None-Match / If-Modified-Since (the tool would get 304s), traceparent, baggage, cache-control and priority. Prefer an allowlist (Accept, Content-Type, plus API-specific X-… headers that pass the secret checks), or at least drop the conditional and tracing headers.

4. Browser upload (frontend/.../connectors/[id]/page.tsx ~69–90). Send only the entries on the connector's origin. Strip credential-named query and body fields before upload, and drop response entirely: the backend doesn't need it.

Smaller

• Path prefix boundary (~305): with base /v1, a request to /v1beta/x is imported as beta/x. Match on a / boundary.
• Object/array defaults get type: 'string' from inferType. Give them object/array, or skip them.

Tests to add

• a nested secret in the body;
• ?key=;
• Ocp-Apim-Subscription-Key;
• a Bearer value in a custom header.

Each should assert that no captured value ends up in tool defaults or headers, and that no env var is created from the HAR.

Happy to re-review as soon as you push. 🎃

[rangarajan19]

Thanks for the detailed review @helpcodeai — pushed fixes for all of it.

Must fix

1. Nested body secrets: har.parser.ts's body handling now recurses into plain objects at any depth (capped at 6). An object is never itself kept as a single default — only scalars/arrays are leaves — so {"user":{"password":"…"}} drops just the nested password and keeps safe siblings (e.g. user.email).
2. Missed credentials: now layers the shared helpers from secret-literals.util.ts (isCredentialHeaderName, isSecretQueryName, findSecretInQueryParam, findSecretLiterals) on top of the issue's regex, plus a local auth-scheme-value check (Bearer …, independent of header name). Verified against all 5 examples you gave — Ocp-Apim-Subscription-Key, X-Access-Key, ?key=AIza…, the X-Firebase-AppCheck JWT, and X-Custom: Bearer … — all now dropped.
3. Header allowlist: switched from "everything except noise/credentials" to an allowlist of Accept + X-* (each still subject to the credential name/value checks). If-None-Match, traceparent, baggage, cache-control, etc. are excluded by default now, not included by default.
4. Browser upload: page.tsx now restricts uploaded entries to the connector's own origin, strips credential-named query/body fields recursively (including nested JSON), and drops response entirely instead of just its body.

Smaller

• Path prefix now matches on a / boundary, so /v1 no longer matches into /v1beta/x.
• Objects are never leaf defaults (always recursed), so they can't be mistyped as string; arrays keep the correct array type and are scrubbed/dropped if any element looks like a credential.

Tests

• Added parser tests for all 4 cases in "Tests to add" (nested secret, ?key=, Ocp-Apim-Subscription-Key, Bearer-in-custom-header) plus the path-boundary case.
• Added a controller-level test (secret-guard.flows.spec.ts) that runs a credential-laden HAR through the actual importTools('har', …) pipeline and asserts addEnvVars is never called and secretsMoved stays undefined — proving the "drop, don't promote" requirement end-to-end, not just inside the parser.
• Full suite: 1347/1347 non-skipped passing, clean typecheck/lint on both packages.

One thing I didn't change, and want your call on: a credential baked directly into a URL path segment (Telegram-style, /bot<TOKEN>/sendMessage) isn't caught by templatePath (it only recognizes numeric/UUID/hex segments as ids, not token-shaped ones), so it still reaches the generic SecretExtractor pass unchanged and gets promoted to a connector env var — the same thing that already happens for a cURL/Postman/JSON import of the same URL shape.

I left it alone because it reads to me as the connector's own routing secret (same value for every caller, not a per-request session artifact), so promoting it seemed consistent with how the other import sources already treat that exact pattern. But your "a HAR import should drop captured values rather than turn them into connector env vars" didn't carve out an exception for this, so I don't want to guess — should this case also be dropped instead of promoted, or is treating it like the other import sources fine here?

Happy to push a follow-up once you weigh in.

[helpcodeai]

Thanks @rangarajan19, I went through the new commits and everything from my review is in: nested bodies are walked and the secret is dropped at any depth, the shared secret helpers are used, headers are an allowlist now, and the browser only uploads entries on the connector's origin with no response. The end-to-end test that checks addEnvVars is never called is a nice touch.

On your question about a token in a path segment (/bot<TOKEN>/sendMessage): keep it the way cURL and Postman handle it. That token is the connector's own secret, not something from the browser session, so promoting it is right. The import guard (secret-extraction.util.ts) already runs on HAR imports through createToolsFromParsed, and it moves the token into an encrypted connector variable instead of leaving it in the tool path. I added a test for exactly that Telegram case so it stays that way.

One small thing I found while reading, and since edits by maintainers are on I just fixed it on your branch (3f06105): form bodies (application/x-www-form-urlencoded) were mapped field by field, but the tool had no bodyEncoding and Content-Type isn't in the header allowlist, so the fields would have gone out as JSON. The parser now sets bodyEncoding: 'form-urlencoded' for those, and reads postData.text when an exporter leaves params out. Tests are in har.parser.spec.ts. I'll merge once CI is green.

[helpcodeai]

Form encoding fixed in 3f06105, everything else from the review is in. Approving.