Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .changeset/dispatcher-per-request-kernel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
"@objectstack/runtime": minor
---

fix(runtime): the HTTP dispatcher serves each request from its OWN resolved kernel — two tenants can no longer swap data sources under each other (#5155)

A host constructs exactly **one** `HttpDispatcher` (`dispatcher-plugin.ts`
`start()`), and every route it serves shares that instance. The kernel a request
resolves to, however, is per request: on a multi-tenant host the injected
`kernelResolver` (ADR-0006) picks a different one per environment.

That per-request answer was being stored on a dispatcher **instance field**,
`this.kernel`, written once per request by `resolveRequestScope()` and then read
by `resolveService()` / `getService()` / `getObjectQL()` /
`getRequestKernelService()` / `announceKernelEvent()` / `getRegisteredAiRoutes()`
— every one of them behind at least one `await`. Node's single thread is no
protection here: what it protects is code that does **not** hold mutable shared
state across an `await`, and this held it across several.

So two interleaved requests on two environments produced this:

1. request A resolves, `this.kernel` = env-1's kernel;
2. A yields at an `await` (session lookup, driver query);
3. request B resolves, `this.kernel` = env-2's kernel;
4. A resumes and resolves `objectql` / `metadata` / `automation` off **env-2**.

One tenant's request reading another tenant's data source — a correctness and
isolation defect, not a performance one. Single-environment deployments were
never affected (`this.kernel === defaultKernel` always, so the write was
idempotent), which is exactly why no local run or CI job ever showed it. It is
now covered by a deterministic interleaving regression test
(`http-dispatcher.multi-tenant-concurrency.test.ts`), which fails on the old
code with request A being served env-2's data.

**The fix: the resolved kernel travels on the request, and every facility that
reads a kernel takes the request explicitly.** `HttpProtocolContext` gains a
`kernel` field, written by `resolveRequestScope()` alongside the
`environmentId` / `dataDriver` / `executionContext` it already writes there.
There is no longer any `this.kernel` to rewrite. An `AsyncLocalStorage` carrier
was deliberately **not** used: it would have reintroduced implicit mutable
ambient context, which is the shape of this bug in a new costume.

Three host-level readers moved to the host kernel explicitly, where they had
been reading whichever tenant resolved most recently: `/ready` (readiness is a
property of the replica), its driver-health probe, and the memoized
single-environment `default-project` lookup.

**Migration — `DomainHandlerDeps` and `ActionExecutionDeps`.** Every
kernel-reading member now takes the request as its **first** parameter. If you
implement or call either contract (both are exported from
`@objectstack/runtime`; nothing in this monorepo or the sibling distributions
did):

- `deps.resolveService(name, envId)` becomes `deps.resolveService(context, name, envId)`
- `deps.getService(name)` becomes `deps.getService(context, name)`
- `deps.getObjectQL(envId)` becomes `deps.getObjectQL(context, envId)`
- `deps.getRequestKernelService(name)` becomes `deps.getRequestKernelService(context, name)`
- `deps.announceKernelEvent(event, payload)` becomes `deps.announceKernelEvent(context, event, payload)`
- `deps.getRegisteredAiRoutes()` becomes `deps.getRegisteredAiRoutes(context)`

`context` is the `HttpProtocolContext` the domain handler already receives. The
same rule applies to the `action-execution` helpers, which take it right after
`deps`: `callData`, `resolveAutomationService`, `dispatchFlowAction`,
`invokeBusinessAction`, `resolveRouteActionDeclaration`.

`HttpDispatcher.getDiscoveryInfo(prefix)` gains an **optional** second argument,
the request context. Callers that serve `/discovery` straight off the host (the
adapters, the dispatcher plugin) need no change and now describe the host kernel
deterministically instead of whichever tenant asked last.

`resolveProjectKernelObjectQL(context)` keeps its direct-caller kernel swap;
the swap is now written onto that context, so it stays visible to the rest of
that request and to nothing else.
2 changes: 1 addition & 1 deletion packages/runtime/src/action-body-identity.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ describe('#3914 — MCP run_action dispatch binds ctx.api and ctx.engine', () =>
};
const ec = { userId: 'user_42', tenantId: 'org_acme', positions: [], permissions: [] };

await invokeBusinessAction(mcpDeps, 'close_case', { recordId: 'case_1' }, {
await invokeBusinessAction(mcpDeps, { request: {} } as any, 'close_case', { recordId: 'case_1' }, {
driver: undefined,
envId: 'platform',
ec,
Expand Down
26 changes: 17 additions & 9 deletions packages/runtime/src/action-execution-calldata-query.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,16 @@

import { describe, it, expect, beforeEach } from 'vitest';
import { callData, type ActionExecutionDeps } from './action-execution.js';
import type { HttpProtocolContext } from './http-dispatcher.js';

const EC = { userId: 'u1', isSystem: false, positions: [], permissions: [] } as any;
/**
* The request `callData` is serving. [#5155] Every service lookup resolves off
* `context.kernel`, so the request has to be named at the call — a fake that
* ignored it would be modelling the shared-field shape this suite's subject no
* longer has.
*/
const REQ = { request: {} } as HttpProtocolContext;

function makeHarness(opts: { withProtocol?: boolean } = {}) {
const finds: any[] = [];
Expand All @@ -39,7 +47,7 @@ function makeHarness(opts: { withProtocol?: boolean } = {}) {
...(protocol ? { protocol } : {}),
};
const deps: ActionExecutionDeps = {
resolveService: (async (name: string) => services[name]) as any,
resolveService: (async (_ctx: HttpProtocolContext, name: string) => services[name]) as any,
getObjectQL: async () => ql,
};
return { deps, finds, findData };
Expand All @@ -58,52 +66,52 @@ describe("callData('query') fallback serves the query it was given (#4386)", ()
offset: 10,
fields: ['id', 'title'],
};
const out = await callData(h.deps, 'query', { object: 'task', query }, undefined, undefined, EC);
const out = await callData(h.deps, REQ, 'query', { object: 'task', query }, undefined, undefined, EC);
expect(h.finds).toHaveLength(1);
expect(h.finds[0]).toMatchObject({ ...query, context: EC });
expect(out.records).toHaveLength(2);
});

it('extracts query fields from bare params when params.query is absent — same source as the protocol path', async () => {
await callData(h.deps, 'query', { object: 'task', where: { status: 'open' }, limit: 3 }, undefined, undefined, EC);
await callData(h.deps, REQ, 'query', { object: 'task', where: { status: 'open' }, limit: 3 }, undefined, undefined, EC);
expect(h.finds[0]).toMatchObject({ where: { status: 'open' }, limit: 3 });
});

it('a caller-supplied context is dropped, never honoured — server-derived only, matching findData', async () => {
await callData(h.deps, 'query', { object: 'task', query: { where: { a: 1 }, context: { isSystem: true } } }, undefined, undefined, EC);
await callData(h.deps, REQ, 'query', { object: 'task', query: { where: { a: 1 }, context: { isSystem: true } } }, undefined, undefined, EC);
expect(h.finds[0].context).toBe(EC);
});

it.each(['sort', 'select', 'skip', 'populate', 'search', 'expand', '$filter'])(
'refuses %s with 501 instead of part-serving — nothing reaches ql.find',
async (key) => {
await expect(
callData(h.deps, 'query', { object: 'task', query: { where: { a: 1 }, [key]: 'x' } }, undefined, undefined, EC),
callData(h.deps, REQ, 'query', { object: 'task', query: { where: { a: 1 }, [key]: 'x' } }, undefined, undefined, EC),
).rejects.toMatchObject({ statusCode: 501 });
expect(h.finds).toHaveLength(0);
},
);

it('names the unservable keys and the served set in the refusal', async () => {
await expect(
callData(h.deps, 'query', { object: 'task', query: { sort: '-x', select: 'id' } }, undefined, undefined, EC),
callData(h.deps, REQ, 'query', { object: 'task', query: { sort: '-x', select: 'id' } }, undefined, undefined, EC),
).rejects.toMatchObject({ message: expect.stringMatching(/'sort', 'select'.*where, fields, orderBy, limit, offset/s) });
});

it('an empty query still lists (the protocol path lists too) — no refusal, no predicate', async () => {
const out = await callData(h.deps, 'query', { object: 'task' }, undefined, undefined, EC);
const out = await callData(h.deps, REQ, 'query', { object: 'task' }, undefined, undefined, EC);
expect(h.finds[0]).toMatchObject({ context: EC });
expect(out.total).toBe(2);
});

it('null-valued keys are withdrawals, not unservable', async () => {
await callData(h.deps, 'query', { object: 'task', query: { sort: null, where: { a: 1 } } }, undefined, undefined, EC);
await callData(h.deps, REQ, 'query', { object: 'task', query: { sort: null, where: { a: 1 } } }, undefined, undefined, EC);
expect(h.finds[0]).toMatchObject({ where: { a: 1 } });
});

it('with the protocol service present the fallback never runs — findData gets the query verbatim, wire spellings included', async () => {
const withP = makeHarness({ withProtocol: true });
await callData(withP.deps, 'query', { object: 'task', query: { sort: '-title', top: 5 } }, undefined, undefined, EC);
await callData(withP.deps, REQ, 'query', { object: 'task', query: { sort: '-title', top: 5 } }, undefined, undefined, EC);
expect(withP.findData).toHaveLength(1);
expect(withP.findData[0].query).toEqual({ sort: '-title', top: 5 });
expect(withP.finds).toHaveLength(0);
Expand Down
50 changes: 32 additions & 18 deletions packages/runtime/src/action-execution.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import { validateActionParams, type ResolvedActionParam } from '@objectstack/spe
import type { ExecutionContext } from '@objectstack/spec/kernel';
import type { IObjectQLEngine, ServiceSlotContract, ServiceSlotContracts } from '@objectstack/spec/contracts';
import { checkApiExposure } from './api-exposure.js';
import type { HttpProtocolContext } from './http-dispatcher.js';
import {
GLOBAL_ACTION_OBJECT_KEY,
actionHandlerObjectKeys,
Expand Down Expand Up @@ -78,21 +79,31 @@ function warnActionParamsOnce(key: string, message: string): void {
* the pattern, alongside `ResolveOptions` in security/resolve-execution-context.
* A lookup facade has to be typed everywhere it is re-declared, or the copy
* that still says `any` becomes the way around all the others.
*
* [#5155] Both lookups take the REQUEST as their first parameter, for the
* reason spelled out on `DomainHandlerDeps` (of which this is the narrow view
* `HttpDispatcher.actionExecutionDeps` hands out): the object is shared by
* every request the host serves, so the kernel to resolve against is the
* request's, never the facade's.
*/
export interface ActionExecutionDeps {
resolveService<K extends keyof ServiceSlotContracts>(name: K, environmentId?: string): Promise<ServiceSlotContract<K> | undefined>;
resolveService(name: string, environmentId?: string): any;
getObjectQL(environmentId?: string): Promise<IObjectQLEngine | null>;
resolveService<K extends keyof ServiceSlotContracts>(context: HttpProtocolContext, name: K, environmentId?: string): Promise<ServiceSlotContract<K> | undefined>;
resolveService(context: HttpProtocolContext, name: string, environmentId?: string): any;
getObjectQL(context: HttpProtocolContext, environmentId?: string): Promise<IObjectQLEngine | null>;
}

/**
* Direct data service dispatch — replaces broker.call('data.*').
* Tries protocol service first (supports expand/populate), falls back to ObjectQL.
*
* @param requestContext - The request being served (#5155). Carries the kernel
* every service lookup below resolves against; see
* {@link HttpProtocolContext.kernel}.
* @param dataDriver - Optional environment-scoped driver to use instead of kernel default
* @param scopeId - Optional project ID for scoped service resolution (SharedProjectPlugin mode)
*/
export async function callData(deps: ActionExecutionDeps,
export async function callData(deps: ActionExecutionDeps,
requestContext: HttpProtocolContext,
action: string,
params: any,
dataDriver?: any,
Expand All @@ -106,7 +117,7 @@ export async function callData(deps: ActionExecutionDeps,
if (!executionContext?.isSystem && params?.object) {
let def: any;
try {
const meta = await deps.resolveService('metadata', scopeId);
const meta = await deps.resolveService(requestContext, 'metadata', scopeId);
def = await (meta as any)?.getObject?.(params.object);
} catch {
def = undefined; // fall open to schema defaults (apiEnabled=true)
Expand All @@ -117,9 +128,9 @@ export async function callData(deps: ActionExecutionDeps,
}
}

const protocol = await deps.resolveService('protocol', scopeId);
const qlService = dataDriver ?? await deps.getObjectQL(scopeId);
const ql = qlService ?? await deps.resolveService('objectql', scopeId);
const protocol = await deps.resolveService(requestContext, 'protocol', scopeId);
const qlService = dataDriver ?? await deps.getObjectQL(requestContext, scopeId);
const ql = qlService ?? await deps.resolveService(requestContext, 'objectql', scopeId);
const qlOpts = executionContext ? { context: executionContext } : undefined;
const findOpts = (extra?: any) => {
const base = qlOpts ? { ...qlOpts } : {};
Expand Down Expand Up @@ -252,8 +263,8 @@ export async function callData(deps: ActionExecutionDeps,
if (!Array.isArray(params.aggregations) || params.aggregations.length === 0) {
throw { statusCode: 400, message: 'aggregate requires at least one aggregation' };
}
const engine = (await deps.getObjectQL(scopeId))
?? await deps.resolveService('objectql', scopeId).catch(() => null);
const engine = (await deps.getObjectQL(requestContext, scopeId))
?? await deps.resolveService(requestContext, 'objectql', scopeId).catch(() => null);
if (engine && typeof engine.aggregate === 'function') {
const rows = await engine.aggregate(
params.object,
Expand Down Expand Up @@ -383,13 +394,13 @@ export function headlessActionTypeError(_deps: ActionExecutionDeps, action: any,
* the single availability probe behind `type: 'flow'` dispatch (both the
* headless-invokability filter and the two invoke paths ask through it).
*/
export async function resolveAutomationService(deps: ActionExecutionDeps, envId?: string): Promise<any | null> {
export async function resolveAutomationService(deps: ActionExecutionDeps, requestContext: HttpProtocolContext, envId?: string): Promise<any | null> {
try {
// [#4127 batch 4] Was `: any`, which voided the gate here. `execute` is
// declared on IAutomationService, so this needed no contract work — only
// for someone to notice, and three grep sweeps over `domains/*.ts` never
// reached this file. The lint rule did.
const svc = await deps.resolveService('automation', envId);
const svc = await deps.resolveService(requestContext, 'automation', envId);
return svc && typeof svc.execute === 'function' ? svc : null;
} catch {
return null; // no automation service on this kernel
Expand Down Expand Up @@ -483,6 +494,7 @@ export function seedFlowActionParams(_deps: ActionExecutionDeps,
* doesn't keep.
*/
export async function dispatchFlowAction(deps: ActionExecutionDeps,
requestContext: HttpProtocolContext,
action: any,
wiring: {
objectName: string;
Expand All @@ -494,7 +506,7 @@ export async function dispatchFlowAction(deps: ActionExecutionDeps,
},
): Promise<any> {
const { objectName, record, params, recordId, ec, envId } = wiring;
const automation = await resolveAutomationService(deps, envId);
const automation = await resolveAutomationService(deps, requestContext, envId);
if (!automation) {
throw new Error(flowActionUnavailableError(action));
}
Expand Down Expand Up @@ -790,7 +802,8 @@ export function buildActionEngineFacade(_deps: ActionExecutionDeps, ql: any, ec?
* attributable and org-scoped. Flow actions differ: the flow engine receives
* the caller's identity below and honours `runAs` (ADR-0049).
*/
export async function invokeBusinessAction(deps: ActionExecutionDeps,
export async function invokeBusinessAction(deps: ActionExecutionDeps,
requestContext: HttpProtocolContext,
name: string,
input: { objectName?: string; recordId?: string; params?: Record<string, unknown> },
wiring: {
Expand Down Expand Up @@ -821,7 +834,7 @@ export async function invokeBusinessAction(deps: ActionExecutionDeps,
if (isSystemObjectName(objectName)) {
throw new Error(`Action '${name}' is on a system object and is not exposed via MCP`);
}
const hasAutomation = Boolean(await resolveAutomationService(deps, envId));
const hasAutomation = Boolean(await resolveAutomationService(deps, requestContext, envId));
if (!isHeadlessInvokableAction(deps, action, hasAutomation)) {
throw new Error(
`Action '${name}' (type='${action?.type ?? 'script'}') cannot be invoked via MCP`,
Expand Down Expand Up @@ -873,15 +886,15 @@ export async function invokeBusinessAction(deps: ActionExecutionDeps,

// ── flow dispatch ── (shared with the REST /actions route, #3915)
if (action.type === 'flow') {
const result = await dispatchFlowAction(deps, action, { objectName, record, params, recordId, ec, envId });
const result = await dispatchFlowAction(deps, requestContext, action, { objectName, record, params, recordId, ec, envId });
return { ok: true, action: action.name, objectName, ...(recordId ? { recordId } : {}), result };
}

// ── script/body dispatch via the engine's executeAction ──
// [#4127] `executeAction` is
// ObjectQL's own surface, outside IDataEngine; `getObjectQL` exists to reach
// exactly that. Closing this needs ObjectQL's contract written, not a cast.
const ql: any = await deps.getObjectQL(envId);
const ql: any = await deps.getObjectQL(requestContext, envId);
if (!ql || typeof ql.executeAction !== 'function') {
throw new Error('Data engine not available for action dispatch');
}
Expand Down Expand Up @@ -1093,6 +1106,7 @@ export async function executeRegisteredAction(_deps: ActionExecutionDeps,
* lookup.
*/
export async function resolveRouteActionDeclaration(deps: ActionExecutionDeps,
requestContext: HttpProtocolContext,
args: { ql: any; objectName: string; actionName: string; envId?: string },
): Promise<{ action: any; obj: any; degraded?: boolean; reason?: string }> {
const { ql, objectName, actionName, envId } = args;
Expand Down Expand Up @@ -1140,7 +1154,7 @@ export async function resolveRouteActionDeclaration(deps: ActionExecutionDeps,
// and belongs in the batch that adds the four undeclared auth members.
// [#4127 batch 4] `loadDiagnosed` is on IMetadataService now, so this
// reads the contract instead of guessing at it.
const meta = await deps.resolveService('metadata', envId);
const meta = await deps.resolveService(requestContext, 'metadata', envId);
if (meta && typeof meta.loadDiagnosed === 'function') {
const diag: any = await meta.loadDiagnosed('action', actionName);
if (diag?.data && ownsRoute(diag.data)) return { action: diag.data, obj };
Expand Down
Loading
Loading