feat(core)!: structural filter language + filterFn rename#110
Merged
Conversation
BREAKING: EndpointCreateOptions.filter is now a serializable structural
clause { dataPath, equals } (or an array of clauses, ANDed) instead of a
predicate function. The old function-based escape hatch is renamed to
filterFn and receives a typed FilterEnvelope instead of unknown.
filter is real, persisted data now (new endpoints.filter column, migration
0005) rather than living only in the process-local callback registry, so
it round-trips through storage and is safe to expose over the admin HTTP
API. filterFn keeps today's registry-based, code-side-only semantics.
There was a problem hiding this comment.
Pull request overview
This PR introduces a breaking change to outbound endpoint filtering by replacing the old function-shaped filter with a persisted, JSON-serializable structural filter ({ dataPath, equals } or an ANDed array), and renames the code-side escape hatch to filterFn with a typed FilterEnvelope. It also bumps the canonical DB schema to version 5 by adding a new endpoints.filter column, wiring it through storage helpers and all SQL adapters, and updates specs/tests/admin/docs accordingly.
Changes:
- Add structural, persisted
filter(round-trips through storage + admin API) and rename predicatefilter→filterFn(typed envelope). - Add DB migration + schema bump to v5 and wire
filterthroughencodeEndpointInsert/decodeEndpointacross all adapters. - Update dispatcher evaluation order (
types/channels/filter/filterFnANDed) plus tests/spec/docs to match.
Reviewed changes
Copilot reviewed 54 out of 54 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| typescript/packages/storage/typeorm/test/conformance.test.ts | Bump expected schema version to 5 for TypeORM conformance. |
| typescript/packages/storage/typeorm/src/storage.ts | Rename callback registry usage to filterFn for TypeORM adapter. |
| typescript/packages/storage/testkit/test/in-memory.test.ts | Bump expected schema version to 5 in shared in-memory testkit run. |
| typescript/packages/storage/testkit/src/index.ts | Ensure filterFn defaults and add round-trip assertions for structural filter. |
| typescript/packages/storage/sqlite/test/conformance.test.ts | Bump expected schema version to 5 for sqlite adapter tests. |
| typescript/packages/storage/sqlite/test/adapter.test.ts | Update migration/schema version assertions to 5. |
| typescript/packages/storage/sqlite/src/storage.ts | Add filter column to sqlite INSERT/UPDATE and move predicate to registry filterFn. |
| typescript/packages/storage/prisma/test/conformance.test.ts | Bump expected schema version to 5 for Prisma adapter tests. |
| typescript/packages/storage/prisma/src/storage.ts | Rename callback registry usage to filterFn for Prisma adapter. |
| typescript/packages/storage/pg/test/testcontainers.test.ts | Bump expected schema version to 5 for PG testcontainers suite. |
| typescript/packages/storage/pg/test/conformance.test.ts | Bump expected schema version to 5 for PG conformance suite. |
| typescript/packages/storage/pg/src/storage.ts | Add filter to hand-written PG UPDATE; rename callback registry to filterFn. |
| typescript/packages/storage/mysql/test/conformance.test.ts | Bump expected schema version to 5 for MySQL conformance suite. |
| typescript/packages/storage/mysql/src/storage.ts | Rename callback registry usage to filterFn for MySQL adapter. |
| typescript/packages/storage/mikro-orm/test/conformance.test.ts | Bump expected schema version to 5 for MikroORM tests. |
| typescript/packages/storage/mikro-orm/src/storage.ts | Rename callback registry usage to filterFn for MikroORM adapter. |
| typescript/packages/storage/kysely/test/conformance.test.ts | Bump expected schema version to 5 for Kysely tests. |
| typescript/packages/storage/kysely/src/storage.ts | Rename callback registry usage to filterFn for Kysely adapter. |
| typescript/packages/storage/helpers/test/helpers.test.ts | Update helpers tests for schema v5, endpoint codec filter round-trip, and filterFn registry rename. |
| typescript/packages/storage/helpers/src/migrations.ts | Add forward-only migration v5 for sqlite/pg/mysql dialect lists. |
| typescript/packages/storage/helpers/src/index.ts | Add filter column encode/decode + rename callback registry field to filterFn (typed envelope). |
| typescript/packages/storage/drizzle/test/conformance.test.ts | Bump expected schema version to 5 for Drizzle tests. |
| typescript/packages/storage/drizzle/src/storage.ts | Rename callback registry usage to filterFn for Drizzle adapter. |
| typescript/packages/core/test/storage.test.ts | Update core storage schema-version expectation to 5; update endpoint seed shapes. |
| typescript/packages/core/test/sender.test.ts | Update endpoint seed shapes to include new filter field. |
| typescript/packages/core/test/review-fixes.test.ts | Update tests to use structural filter + add filter: null where needed. |
| typescript/packages/core/test/retry.test.ts | Update endpoint seed shapes to include new filter field. |
| typescript/packages/core/test/keys-replay.test.ts | Update endpoint seed shapes to include new filter field. |
| typescript/packages/core/test/jwks-publish.test.ts | Update endpoint seed shapes to include new filter field. |
| typescript/packages/core/test/dispatcher.test.ts | Add structural filter scenario coverage; rename predicate usage to filterFn; verify read-shape round-trip. |
| typescript/packages/core/test/config-audit.test.ts | Update endpoint seed shapes to include new filter field. |
| typescript/packages/core/src/storage/types.ts | Make EndpointRecord.filter persisted structural data; add filterFn callback field. |
| typescript/packages/core/src/storage/memory/adapter.ts | Bump in-memory schema version to 5; store filterFn separately from structural filter. |
| typescript/packages/core/src/sender/endpoint/crud.ts | Add filter to public endpoint read shape + thread through create/update; support filterFn. |
| typescript/packages/core/src/sender/dispatcher/http-dispatcher.ts | Read endpoint.filterFn directly instead of function-checking endpoint.filter. |
| typescript/packages/core/src/sender/dispatcher/filter-transform.ts | Implement structural filter matching and typed envelope for predicate/transform evaluation. |
| typescript/packages/core/src/outbound.ts | Define Json, StructuralFilter*, FilterEnvelope; change create options to filter + filterFn; include filter on read shape. |
| typescript/packages/core/src/index.ts | Re-export new filter-related public types. |
| typescript/packages/admin/test/admin-router.test.ts | Add test proving structural filter crosses admin create/get. |
| typescript/packages/admin/src/index.ts | Accept and normalize structural filter in admin request bodies. |
| specs/db-schema/0005_endpoint_structural_filter.sql | Add canonical migration 0005: endpoints.filter column + schema_version=5. |
| openspec/specs/storage-layer/spec.md | Update storage-layer spec to require endpoints.filter column and add scenario. |
| openspec/specs/filtering-transformation/spec.md | Add structural filtering requirement; rename predicate filter to filterFn and type envelope. |
| openspec/specs/endpoint-management/spec.md | Update endpoint CRUD requirement to include filter round-trip and filterFn as code-only. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/tasks.md | Archived OpenSpec change tasks/checklist for this capability update. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/specs/storage-layer/spec.md | Archived OpenSpec delta for storage-layer spec update. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/specs/filtering-transformation/spec.md | Archived OpenSpec delta for filtering/transformation spec update. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/specs/endpoint-management/spec.md | Archived OpenSpec delta for endpoint-management spec update. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/proposal.md | Archived OpenSpec proposal explaining the change rationale and scope. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/language-impact.md | Archived OpenSpec language impact notes for ports and contract/port-specific split. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/db-schema-delta.sql | Archived DB schema delta for the OpenSpec change. |
| openspec/changes/archive/2026-07-09-add-structural-filter-language/.openspec.yaml | Archived OpenSpec change metadata. |
| docs/content/docs/outbound/endpoints.mdx | Document structural filter and filterFn escape hatch semantics. |
| docs/content/docs/outbound/admin.mdx | Document that structural filter is admin-API configurable; code-only fields aren’t. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Three bugs from review:
- jsonDeepEqual recursed over the untrusted value's own keys rather than
the trusted equals literal's, so a cyclic value could in principle
drive unbounded recursion. Recursion now follows equals's (finite,
config-supplied) structure, and a WeakSet guards against revisiting
the same value reference as defense in depth.
- Non-plain objects (Date, class instances) compared as empty plain
objects via Object.keys, so e.g. a Date value could incorrectly
deep-equal {}. Now rejected via a real plain-object check instead of
an unchecked `as Json` cast.
- resolveDataPath treated empty path segments (e.g. "a..b") as real
keys instead of treating the path as malformed (no match).
Also fixed a stale doc comment claiming both filterFn and transform
receive a typed FilterEnvelope; transform's public type is unchanged.
…-language # Conflicts: # typescript/packages/core/src/index.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EndpointCreateOptions.filteris now a serializable structural clause —{ dataPath, equals }, or an array of clauses ANDed together — matching when the event'sdataatdataPathdeep-equalsequals. BREAKING: this replaces the old(event) => booleanpredicate.filterFn, and now receives a typedFilterEnvelope({ type, data, channels?, timestamp? }) instead ofunknown.filteris real, persisted data: newendpoints.filtercolumn (migration0005), wired throughencodeEndpointInsert/decodeEndpointso all 8 SQL storage adapters pick it up.filternow round-trips on theEndpointread shape and crosses the admin HTTP API.filterFnkeeps today's semantics exactly — process-local callback registry on non-memory storage, ephemeral across restarts, absent from the read shape and the admin API.types/channels/filter/filterFnall compose as AND at dispatch time.Closes #95.
Capabilities
filtering-transformation: ADDED "Structural filter matches a data path"; MODIFIED "Predicate filter" (→filterFn+ typed envelope) and "Filter and transform errors fail closed" (→filterFn)endpoint-management: MODIFIED "Endpoint CRUD" (filternow round-trips;filterFnreplacesfilterin the function-shaped/absent-from-read-shape list)storage-layer: MODIFIED "Schema is a fixed set of canonical tables" (endpoints.filtercolumn)DB schema impact
New forward-only migration
specs/db-schema/0005_endpoint_structural_filter.sql—ALTER TABLE endpoints ADD COLUMN filter jsonb(SQLite:TEXT), bumpsschema_versionto 5. Applied centrally via@postel/storage-helpers'encodeEndpointInsert/decodeEndpoint, so all 8 SQL adapters (pg,mysql,sqlite,kysely,drizzle,typeorm,prisma,mikro-orm) pick it up;pg/mysql/sqliteadditionally needed the column added to their hand-writtenUPDATEstatements (the other five derive their update column list dynamically).Test plan
dispatcher.test.ts(single clause match/mismatch, nested path, ANDed array, missing path, read-shape round-trip)filterFn; updated "Function-shaped options stay off the read shape" to checkfilterFn(notfilter) is absentstorage-testkit's sharedEndpoint CRUDconformance test now exercisesfilterround-tripping — runs against all 8 real adapters, not just in-memory@postel/admin— new test provingfiltercrosses the admin HTTP API on create + getpnpm typecheck/pnpm test/pnpm lint/pnpm buildgreen across all 27 packagesmise run check:allgreen after archiving (spec-drift, spec:validate, spec:schema-validate)