feat(functions-compiler): plan, size and split an app's function shards

packages/functions-compiler/README.md

@@ -17,52 +17,91 @@ Two consumers share this one engine:
 Internal to Base44 — published **public** so apper's bundler service can install
 it, but it is not a supported public API: the CLI bundles it at build time so
 end users never install it, and it carries no compatibility promise to anyone
-outside this repo. It compiles functions and nothing else: shard planning, size
-splitting, artifact writing, version creation and deploy all live above it.
+outside this repo. Compilation is the whole of its job, and that now includes
+source assembly, shard planning, size measurement and splitting. Artifact
+writing, version creation, upload and deploy live above it.
 
 The package carries no credentials and reads no configuration of its own. It
 names the environment variables and headers the generated worker will use at
 runtime (`BASE44_*`, `X-Base44-*`) but holds none of their values, and the code
 it ships is the same code already compiled into every deployed user worker.
 
-## What is missing: shards, and whole-app CFW bundles
+## Shards and whole-app builds
 
-Today this package compiles **one module per call**. It does not decide which
-functions belong in which module, how large the result may be, or what to do
-when it is too large. That work is still Python in apper's
-`backend/app/cloudflare_functions/` and moves here next — it is what turns
-"compile this set of sources" into "produce the deployable Cloudflare Workers
-bundles for this app".
+This package turns an app's whole backend function set into deployable Cloudflare
+Workers: it assembles each function's sources, groups them into shards, compiles
+each shard, measures it against Cloudflare's ceilings, and halves any shard whose
+module is over one. `compileFunctionShards` is the entry point, and unlike
+`bundleApp` it refuses a partial result — a module missing a handler is a broken
+app, not a smaller success.
 
-| Missing piece | Where it lives today | What it has to do |
+Each piece came from apper, and the provenance is worth keeping:
+
+| Piece | Where it came from | Here |
 |---|---|---|
-| Source assembly | `function_bundle.py` — `cfw_bundle_input`, `collect_reachable_backend_files` | Turn a function's directory plus the shared files it reaches into the `entry` + `files` this package takes, keeping the flat single-file case and the refusal to escape into the frontend tree |
-| Fresh shard planning | `shard_planning.py` — `full_repartition`, `target_shard_count` | Group an app's functions into shards deterministically, and refuse a set that exceeds the supplied product capacity |
-| Size measurement | `cloudflare_wfp_runtime.py` — `measure_bundle_bytes`, `judge_bundle_size` | Raw UTF-8 bytes and level-6 gzip, against Cloudflare's 64 MiB uncompressed limit and our compressed cap |
-| Split on overflow | `cloudflare_wfp_runtime.py` — `_build_shard_with_split` | Halve an oversized multi-function shard deterministically and recompile; fail the build when one function alone is too big |
-| Whole-build validation | apper PR #23460 | Reject a partial result: every declared function in exactly one successful shard, or no build at all |
-
-Only the **fresh-build** slice comes across. Everything that remembers a
-previous deploy stays in apper: incremental shard reuse (it needs the previous
-deployment map), the per-app `shard_size_override` ratchet, entitlement and
-settings reads, provider upload, binding resolution and secret delivery. Policy
-numbers — shard size, shard count, the gzip cap, whether the gate is enforced —
-arrive as inputs; the package never reads them itself.
-
-Two details from the Python that must survive the port, both verified against
-the current code:
-
-- The single-shard path builds in caller order while the multi-shard path sorts
-  by name. Function order changes the emitted bytes, so both branches carry
-  over as they are — normalising them is a byte change dressed as a cleanup.
-- Capacity is judged at the *global* shard size while packing may use a smaller
-  ratcheted one, so a legal plan can hold more shards than `max_shards`. A
-  final "shard count ≤ max_shards" assertion would lock out apps that deploy
-  fine today.
-
-Also deliberately out of scope for the new lane: Deno deployment targets,
-existing-Worker reuse, incremental deploy state, and actors — though the engine
-keeps its actor support for the legacy service that still uses it.
+| Source assembly | `function_bundle.py` — `cfw_bundle_input`, `collect_reachable_backend_files` | `src/assembly.ts` |
+| Fresh shard planning | `shard_planning.py` — `full_repartition`, `target_shard_count` | `src/shards/plan.ts` |
+| Size measurement | `cloudflare_wfp_runtime.py` — `measure_bundle_bytes`, `judge_bundle_size` | `src/shards/size.ts` |
+| Split on overflow | `cloudflare_wfp_runtime.py` — `_build_shard_with_split` | `src/shards/build.ts` |
+| Whole-build validation | apper PR #23460 | `src/shards/build.ts` |
+
+Policy numbers — shard size, shard count, the compressed cap — arrive as inputs.
+The package reads no settings and no feature flags, and the two wrapper flags
+(`postResponseTelemetry`, `runtimeSecrets`) are handed to it, because both change
+the emitted bytes and only the platform knows their value for an app.
+
+Deliberately not here: Deno deployment targets, existing-Worker reuse, provider
+upload, binding resolution, secret delivery, and incremental shard reuse — every
+version is built from scratch, so nothing here remembers a previous deploy. The
+engine keeps its actor support for the legacy service that still uses it.
+
+### What a bundle says about itself
+
+The first line of a compiled shard is its own manifest:
+
+```js
+//!b44:1 {"functions":["cleanupForgottenDepartures","health","sendReminder"],"telemetry":false,"runtimeSecrets":false,"compiler":"0.1.0"}
+```
+
+`//!b44:<format>` is a fixed sentinel, so `head -1` on a script pulled from
+Cloudflare answers "what is in this?" without executing or parsing anything, and
+the payload is JSON so a tool parses it in one call. Only the app path emits it;
+the legacy single-function `bundle()` stays bannerless, which keeps that lane
+byte-comparable with the engine apper still runs.
+
+`functions` is sorted whatever order the shard was built in. `telemetry` and
+`runtimeSecrets` are there because they change the emitted bytes and a deploy has
+to pair its secrets delivery with them. `compiler` is this package's version.
+
+Nothing volatile may be added: a timestamp or build id would re-mint a version
+for unchanged code, the app id would make the same functions compile differently
+per app, and the shard's position would make two identical shards differ.
+
+### Reproducibility is a contract, not a nicety
+
+A version's identity is the hash of the **compiled artifacts**, never of the
+sources. Anything that shifts the emitted bytes therefore mints a new version of
+code that did not change, and all of these do:
+
+- the version of this package, and the depth of the `node_modules` it was built
+  against — esbuild writes each vendored chunk's relative path into the minified
+  output, so building one directory shallower changes every user Worker's bytes;
+- the order the caller hands functions over in, for a single shard: that path
+  builds in caller order, while a multi-shard plan sorts by name. Both branches
+  are as apper has them, and `shard-build.e2e.test.ts` pins the difference;
+- either wrapper flag.
+
+Two behaviours differ from apper's Python on purpose, and both are tested:
+
+- **An unparseable source.** apper reads each file on its own, so a file it
+  cannot parse contributes no edges while the rest of the set still assembles.
+  esbuild's walk is one build over the whole graph, so anything unparseable in it
+  rejects — the fallback is the flat single-file submission, and the compiler then
+  reports the error itself.
+- **Compressed size.** Node's gzip reads about 0.7% heavier than Python's on
+  identical input: 43,057 bytes against 42,765 on a real 122,324-byte module. The
+  direction is the safe one, since this lane refuses slightly earlier than the
+  service would.
 
 ## Using it
 

packages/functions-compiler/src/assembly.ts

@@ -0,0 +1,201 @@
+/**
+ * Turn a backend function into the `{entry, files}` the compiler takes.
+ *
+ * A function may import a helper beside it or a module shared across functions
+ * (`../../shared/x.ts`). The compiler resolves relative imports by exact match
+ * inside `files`, so anything the function reaches has to be submitted with it —
+ * otherwise the import has no target and the build fails with "can't reach
+ * outside the function".
+ *
+ * Ported from apper's `backend/app/cloudflare_functions/function_bundle.py`.
+ * That walks the import graph with a tree-sitter parse, as a stand-in for what
+ * the bundler would see; here esbuild does the walk, so it is not a stand-in —
+ * the file set is what the compiler itself reaches. Verified against the Python
+ * fixtures in assembly.test.ts.
+ */
+
+import path from "node:path";
+import { build } from "esbuild";
+
+const NAMESPACE = "base44-reachability";
+
+/** The flat shape a single-file function has always been submitted as. Keeping
+ *  it means those functions compile to the same bytes they do today. */
+const FLAT_ENTRY = "main.ts";
+
+export interface BundleInput {
+  entry: string;
+  files: Record<string, string>;
+}
+
+function loaderFor(filePath: string) {
+  if (/\.(ts|mts|cts)$/.test(filePath)) return "ts" as const;
+  if (filePath.endsWith(".tsx")) return "tsx" as const;
+  if (filePath.endsWith(".jsx")) return "jsx" as const;
+  if (filePath.endsWith(".json")) return "json" as const;
+  return "js" as const;
+}
+
+function resolveRelative(
+  importer: string,
+  spec: string,
+  files: Record<string, string>,
+): string | null {
+  const dir = importer.includes("/")
+    ? importer.slice(0, importer.lastIndexOf("/"))
+    : "";
+  const resolved = path.posix.normalize(path.posix.join(dir, spec));
+  return resolved in files ? resolved : null;
+}
+
+/** esbuild's own account of what the entry reaches: the metafile inputs, keyed
+ *  by the paths the plugin resolved. Its own function so a failure can be
+ *  handled by the caller. */
+async function walkInputs(
+  entryPath: string,
+  backendFiles: Record<string, string>,
+): Promise<Record<string, unknown>> {
+  const result = await build({
+    entryPoints: [entryPath],
+    bundle: true,
+    write: false,
+    metafile: true,
+    logLevel: "silent",
+    // Only the metafile is read, never the output, so neither of these can
+    // change what the compiler emits. They decide what the walk ACCEPTS: the
+    // default iife format rejects top-level await, which a Deno function may
+    // legitimately use, and the walk would then reject a file the compiler
+    // itself compiles fine.
+    format: "esm",
+    platform: "neutral",
+    // Every path here is a project path served from memory and nothing resolves
+    // to the filesystem, so this only has to be a valid absolute path — and
+    // `path.sep` is not one on Windows, where esbuild refuses "\\".
+    absWorkingDir: process.cwd(),
+    // TypeScript drops an import whose bindings go unused, which would hide a
+    // file the function really does pull in. Keep every import as written.
+    tsconfigRaw: { compilerOptions: { verbatimModuleSyntax: true } },
+    plugins: [
+      {
+        name: "base44-reachability",
+        setup(pluginBuild) {
+          pluginBuild.onResolve({ filter: /.*/ }, (args) => {
+            if (args.kind === "entry-point") {
+              return { path: entryPath, namespace: NAMESPACE };
+            }
+            if (args.path.startsWith("./") || args.path.startsWith("../")) {
+              const target = resolveRelative(
+                args.importer,
+                args.path,
+                backendFiles,
+              );
+              if (target) return { path: target, namespace: NAMESPACE };
+            }
+            return { path: args.path, external: true };
+          });
+
+          pluginBuild.onLoad(
+            { filter: /.*/, namespace: NAMESPACE },
+            (args) => ({
+              contents: backendFiles[args.path],
+              loader: loaderFor(args.path),
+            }),
+          );
+        },
+      },
+    ],
+  });
+  return result.metafile.inputs;
+}
+
+/** Did this build fail on the user's code, rather than on how the walk was
+ *  set up? Every message esbuild raises about a source it loaded carries a
+ *  location in this plugin's namespace; a configuration fault carries no
+ *  location at all (verified against both shapes). */
+function isUserSourceFailure(error: unknown): boolean {
+  const errors = (
+    error as { errors?: { location?: { file?: string } | null }[] }
+  ).errors;
+  return (
+    Array.isArray(errors) &&
+    errors.length > 0 &&
+    errors.every((e) => e.location?.file?.startsWith(`${NAMESPACE}:`) === true)
+  );
+}
+
+/**
+ * Every backend file reachable from `entryPath` through `./` and `../` imports,
+ * including the entry. A specifier with no target in `backendFiles` — a typo, or
+ * a frontend file the caller deliberately excluded — is left out, so it stays
+ * forbidden by the compiler rather than being resolved here. `npm:`, `jsr:`,
+ * `node:` and bare specifiers are not edges into the project and are ignored.
+ *
+ * A source the walk cannot parse yields the entry alone, so the function
+ * compiles as the flat single-file submission and the compiler reports the
+ * error itself.
+ */
+export async function collectReachableFiles(
+  entryPath: string,
+  backendFiles: Record<string, string>,
+): Promise<Record<string, string>> {
+  if (!(entryPath in backendFiles)) {
+    throw new Error(`entry "${entryPath}" is not among the backend files`);
+  }
+
+  let inputs: Record<string, unknown>;
+  try {
+    inputs = await walkInputs(entryPath, backendFiles);
+  } catch (e) {
+    // apper's walk cannot fail this way: it reads each file on its own, so a
+    // file it cannot parse contributes no edges and the rest of the set still
+    // assembles. esbuild's walk is all-or-nothing — one unparseable source
+    // anywhere in the graph rejects here. Falling back to the entry alone keeps
+    // the flat submission a single-file function has always had, and the compile
+    // then fails with the compiler's own diagnostic, which the builder agent can
+    // act on. An exception out of assembly has nowhere to be reported at all.
+    //
+    // Only for the user's own sources, though. A fault in how this walk is
+    // configured also arrives here, and degrading it to a flat submission
+    // reports OUR bug as the user's unresolved import — which is how the format
+    // and working-directory defects above stayed invisible.
+    if (!isUserSourceFailure(e)) throw e;
+    return { [entryPath]: backendFiles[entryPath] };
+  }
+
+  const reached: Record<string, string> = {};
+  for (const input of Object.keys(inputs)) {
+    const filePath = input.startsWith(`${NAMESPACE}:`)
+      ? input.slice(NAMESPACE.length + 1)
+      : input;
+    if (filePath in backendFiles) reached[filePath] = backendFiles[filePath];
+  }
+  return reached;
+}
+
+/**
+ * The `{entry, files}` to compile for one function.
+ *
+ * A function that reaches nothing beyond its own entry keeps the flat
+ * `main.ts` submission, byte-identical to how it compiles today. One that
+ * reaches a helper or a shared module gets its real project path as the entry,
+ * so `../../shared/x.ts` resolves against the files beside it.
+ *
+ * `backendFiles` must hold the backend tree only. A frontend path left in it
+ * would resolve, and a function would silently bundle frontend code.
+ */
+export async function cfwBundleInput(
+  entryPath: string,
+  entryContent: string,
+  backendFiles: Record<string, string>,
+): Promise<BundleInput> {
+  const files = await collectReachableFiles(entryPath, {
+    ...backendFiles,
+    [entryPath]: entryContent,
+  });
+  const reachedOnlyTheEntry =
+    Object.keys(files).length === 1 && entryPath in files;
+  if (reachedOnlyTheEntry) {
+    return { entry: FLAT_ENTRY, files: { [FLAT_ENTRY]: entryContent } };
+  }
+  return { entry: entryPath, files };
+}

packages/functions-compiler/src/deno-bundle.ts

@@ -73,6 +73,9 @@ export async function bundleToModule(
         // (npm/Emscripten glue), leaving loader-bound CJS locals intact.
         define: { __dirname: '"/"', __filename: '"/index.js"' },
         minify: true,
+        // Prepended verbatim and not minified away, which is the point: it is
+        // the one place a compiled module says what is inside it.
+        ...(prepared.banner ? { banner: { js: prepared.banner } } : {}),
         sourcemap: false,
         // Errors are surfaced structurally (return value / BuildFailure); keep
         // esbuild from dumping diagnostics to the service's stderr.

packages/functions-compiler/src/index.ts

@@ -2,6 +2,8 @@
 // src/ is either an internal engine module or a compile-time asset read as text
 // by the esbuild plugins.
 
+export type { BundleInput } from "./assembly.js";
+export { cfwBundleInput, collectReachableFiles } from "./assembly.js";
 export type {
   AppErrorClassification,
   AppFunctionStatus,
@@ -30,6 +32,28 @@ export { DenoCompatError } from "./errors.js";
 export { createGuardedFetch, installFetchGuard } from "./fetch-guard.js";
 export type { Field, Level, LogSink } from "./log.js";
 export { setLogSink } from "./log.js";
+export type {
+  CompiledShard,
+  ShardBuildFailure,
+  ShardBuildResult,
+} from "./shards/build.js";
+export { compileFunctionShards } from "./shards/build.js";
+export type { ShardPolicy } from "./shards/plan.js";
+export {
+  assertWithinCapacity,
+  planFreshShards,
+  ShardCapacityError,
+  targetShardCount,
+} from "./shards/plan.js";
+export type { BundleSize, SizeVerdict } from "./shards/size.js";
+export {
+  BUNDLE_GZIP_LEVEL,
+  judgeBundleSize,
+  measureBundleBytes,
+  WORKER_RAW_SIZE_CEILING_BYTES,
+  workerGzipCapBreach,
+  workerRawSizeBreach,
+} from "./shards/size.js";
 export { STATIC_EGRESS_ARTIFACT_MARKER } from "./static-egress-marker.js";
 export type { CompilerTracer } from "./tracing.js";
 export { setCompilerTracer } from "./tracing.js";