Skip to content

Rebuild only what a change affects in the XDK build - #778

Open
lagergren wants to merge 10 commits into
masterfrom
lagergren/deterministic-xtc
Open

lagergren wants to merge 10 commits into
masterfrom
lagergren/deterministic-xtc

Conversation

@lagergren

@lagergren lagergren commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

A commit that changes no source, a comment in a library, or a change inside one method rebuilt and retested most of the XDK. With these commits a build reruns only what the change affects. Part of #665: the compiler no longer stamps the time of the compile; the absolute source directory and file times remain.

Note: The ModelApi is a well understood and well tested concept that only has existed in the language server as a kotlin class for quite a while. I just ported that to Java and put it in Javatools so that the Gradle Plugin can also utilize it for its dependency analysis for the XtcCompile tasks.

What changes

  1. Make the compiler's output depend only on its input

    • Modules are stamped with the epoch instead of the time of the compile.
      • Nothing reads that timestamp. The launchers, the runtime and platform decide what's newer by the files' modification times.
      • But a changing timestamp made every compile's output new. So no compile against a module, and no test of it, could ever be reused, either as up to date or from the build cache. That was the main reason building the XDK kept redoing everything.
    • xtc build --timestamp stays available for compatibility, although nothing uses it. It takes now, or an ISO-8601 instant such as a release commit's time. The module description shows a timestamp only when one was set.
    • The Gradle plugin passes --timestamp only if a build sets xtcCompile { timestamp = ... }. The XDK build must not set it: it would bring back the rebuilds and the build cache misses this PR removes.
    • Each method numbers its own _ variables, instead of using a counter shared by the whole JVM.
    • The same source now always compiles to the same bytes. A change that doesn't change a module's bytes stops at that module, and nothing compiled against it is rebuilt.
    • DeterministicOutputTest compiles a module twice in one JVM and compares the bytes.
  2. Stop a commit that changes no source from rebuilding the XDK

    • javatools.jar records the git commit and status in build-info.properties for xtc --version, and every XDK compile and test took the jar as an input.
    • So every commit, branch switch or rebase rebuilt and retested the whole XDK, and the build cache could never reuse a result from another commit.
    • The plugin's tasks now take the jar as a @Classpath, and runtime classpath normalization ignores just git.commit and git.status. A real change to javatools still reruns everything.
    • The deliverables still carry the commit: after an empty commit, javatools.jar, the installed XDK and the distribution zip all have the new one, and xtc build --version prints it. Normalization only changes how Gradle compares the jar, never what is in it.
  3. Stop a commit from rerunning the run and test tasks of XDK consumers

    • XtcRunTask had a duplicate input: the whole unpacked XDK directory, javatools.jar included, compared as plain files.
    • That made manualTests' tests rerun on every commit. The modules and jars are still tracked by the existing inputs.
  4. Run ModuleInfoTest on its own, with the XDK sources it reads

    • All javatools tests took the lib_net, lib_json and lib_web sources as inputs, for the one test class that reads them.
    • That class now runs in its own moduleInfoTest task, which check runs.
  5. Define the API of a module: what a module that imports it compiles against

    • ModuleApi writes a module without its methods' code, their source text, its timestamp and its source directory, followed by whether each method is abstract, native or has code.
    • The compiler reads nothing else from the modules it compiles against, so an importer compiles to the same bytes against two versions with the same API.
    • The tests check both directions:
      • ModuleApiTest: a change inside a method body keeps the API and the importer's bytes, while a changed signature, an interface method that gains a default body, a changed constant and a new method each change the API.
      • ModuleApiStripTest: no method of the XDK's modules keeps code or source text.
  6. Compile a module again only when the API of a module it imports changes

    • A compile task took the whole files of the modules it imports as input. So any change to ecstasy, even one string inside a method, compiled the 21 modules that import it again, and each came out byte for byte the same: 117 s of the 170 s of task time in that build.
    • An artifact transform now turns each imported module into its API, and the compile task takes those as input in place of the modules.
      • The compile still reads the modules themselves.
      • Tests, runs and the distribution still take the whole modules, since they execute or ship the code.
    • The transform loads javatools for each run and closes the class loader afterwards, so nothing holds open a jar that the build rewrites.
    • Wherever it can't compute an API, it copies the file, so the compile depends on all of it, as before. That covers: no javatools, a javatools without ModuleApi, a file that isn't a readable module, and a Gradle JVM too old to load javatools (the compile itself runs in a JVM of its own). The ConfigurationCacheCompatibilityTest failure on this PR's first run was the no-javatools case.
    • The API files are named <module>.api and live only in Gradle's transform cache. Nothing loads them, and they aren't valid module files: the module format is unchanged.

Build times

./gradlew build after a full build, with the lang build off and manualTests not attached, as they are by default. The master column is the last commit before this PR. The changes inside a method were measured on this branch before and after commit 6.

Change master with this PR
Nothing 0.8 s 0.6 s
A commit, branch switch or rebase that changes no source 70 s: 24 compiles, 25 test tasks 9–11 s: no compiles or tests, only the jar and the distribution
Comment in lib_ecstasy 58 s: 24 compiles, 25 test tasks 7 s: 1 compile
Comment in lib_json 40 s: 14 compiles, 15 test tasks 4 s: 1 compile and moduleInfoTest
A string changed inside an ecstasy method 22 compiles, 25 test tasks (59 s before commit 6) 37 s: 1 compile, and the tests
A change to ecstasy's API (a new class) 22 compiles and the tests the same: every importer compiles again
A real change in javatools reruns everything the same

With manualTests attached, its tests no longer rerun after a commit or a comment. They rerun when a library they use really changes. This was measured on another branch with commits 1 to 4.

Checked in a new checkout with an empty GRADLE_USER_HOME

Step Compiles Tests Time
Full build 22 29 168 s, including downloads
Nothing 0 0 1 s
Empty commit 0 0 9 s
String inside an ecstasy method 1 26 38 s; only ecstasy.xtc changed
Revert it 0 0 5 s, from the build cache
New class in ecstasy 22 26 66 s
Revert it 0 0 2 s, from the build cache
clean, then build 0 0 7 s: 92 tasks from the build cache

After the round trip through the cache, all 69 outputs were byte for byte what the build had made: every module, the javatools jar, the distribution zip and the installed XDK.

Things to know

  • The plugin now sets a normalization rule in every project that applies it. Their own classpath inputs also ignore git.commit and git.status in a build-info.properties at the root of a classpath entry. It's narrow, but third-party projects get it too.
  • The first build of an existing checkout after this rebuilds everything once, because the plugin changed.
  • CI keeps the build cache between runs. This PR's own CI can't reuse master's XTC compiles and tests, because it changes the plugin and the compiler. Rerun against master's cache, it took 516 s instead of 825 s: the build-logic, Kotlin and javatools_utils tasks came from the cache, and a lang marker skipped the lang validation. Once this is merged, a pull request that doesn't change javatools, the plugin or the XDK sources can reuse the XTC compiles and tests too.
  • Not done here:

Testing

  • ./gradlew build passes on this branch, and in a new checkout with an empty GRADLE_USER_HOME.
  • The new tests ran with nothing skipped: DeterministicOutputTest, CompilerCounterAtomicTest, moduleInfoTest, ModuleApiTest and ModuleApiStripTest.
  • Both new API tests fail when the method source is kept in the API.
  • ./gradlew spotlessCheck is clean.
  • Run locally the way CI runs it: the build step with CI, RUN_INTEGRATION_TESTS and manualTests attached (all 62 plugin tests, none skipped), the manualTests CI tasks, and xtc-app-template greet against the published Maven repository.

Compiling the same source twice gave different module bytes, for two reasons:

- Each module was stamped with the wall-clock time of its compile. Nothing reads that
  timestamp, but it made every compile's output new. The compiler now stamps the epoch,
  as FileStructure already allows for reproducible output.
- A variable named _ got a name numbered by a counter shared by every compile in the
  JVM, so a compile's names depended on what else the JVM had compiled before it. A
  method now numbers within itself, as a property outside a method already did.

The timestamp is what made the build's compiles cascade: any recompile of a module,
even for a comment, gave new bytes, so every module compiled against it was out of date,
and so were their tests and the javatools tests that use the XDK. Now a change that
does not change a module's bytes stops at that module. A trailing comment in
lib_ecstasy costs one compile (7s) instead of a rebuild of all 22 XDK modules and their
tests (96s). The counter matters for compiles that share a JVM: the embedding API, the
language server and compiles of several modules in one process.
javatools' build-info.properties records the commit and branch it was built from, for
xtc --version. That file is in the javatools jar, which every task that compiles or runs
modules takes as an input, so each commit, branch switch or rebase made every XDK compile,
every XDK test and the javatools tests out of date, and no build cache entry from another
commit could be used: 106s for a build after a commit that changed nothing.

Nothing that compiles or runs modules reads the commit or the branch, and the version test
compares the output with the same file. So the XTC plugin's tasks now take the javatools
jar as a classpath, which it is, and the plugin's projects and javatools compare classpaths
without those two properties, with Gradle's runtime classpath normalization. A different
commit now rebuilds only the jar and the distribution, which should carry it: 11s. The
version numbers in the same file are still compared, and a change to any class or other
resource in the jar still reruns everything.
XtcRunTask took the unpacked XDK directory as an input of its own, besides the XDK's modules
and its jars, which XtcLauncherTask already takes. The directory holds only those, so it
tracked nothing more, but it compared javatools.jar as a plain file, with the commit and
branch that its build-info.properties records, which the classpath comparison ignores. So
every commit reran the tests of a project that consumes the XDK, such as manualTests (9s).
The task still depends on the task that unpacks the XDK, which it names explicitly.
The javatools tests took the sources of lib_net, lib_json and lib_web as inputs, because
ModuleInfoTest parses them, so any edit to those libraries, even a comment, reran all of the
tests (62s) for the sake of one class that takes 0.2s. ModuleInfoTest now runs in a task of
its own, moduleInfoTest, which check runs, with those sources as its inputs. A comment in
lib_json now costs a build of 8s instead of 70s.
…ainst

ModuleApi writes a module file without its methods' code, the source text
of their bodies, its timestamp and its source directory, followed by
whether each method is abstract, native or has code. The compiler reads
nothing else from the modules it compiles against, so a module compiles
to the same bytes against two versions of an import with the same API:
a build can then skip compiling the importers after a change inside a
method body.

ModuleApiTest compiles a library in two versions that differ only inside
a method body, and checks that their APIs and an importer's bytes are the
same; and that a changed signature, an interface method that gains a
default body, a changed constant and a new method each change the API.
ModuleApiStripTest checks that no method of the XDK's modules keeps code
or source text in the API.
A compile task took the whole files of the modules it imports as input,
so any change to ecstasy, such as one string inside a method body,
compiled all 21 modules that import it again, though each came out byte
for byte the same: 117 s of the 170 s of task time in that build.

An artifact transform now turns each imported module into its API
(ModuleApi), and a compile task takes those as its input in place of the
modules. The compile still reads the modules themselves, and the tasks
that run or ship them, tests and the distribution among them, still take
the whole modules as input. The transform loads javatools for each run and
closes the class loader after it, so that no class loader keeps a jar
open that the build rewrites; with a javatools that has no ModuleApi it
copies the modules, which keeps the old behavior.

The same body change in ecstasy now compiles 1 module instead of 22 and
reruns the tests: 37 s instead of 59 s.
… one

ConfigurationCacheCompatibilityTest failed on CI: its project depends on
a file that is not a module and has no javatools, and the transform failed
the build where the compile had only compared the file. Wherever the
transform cannot compute an API, it now copies the file, so that a compile
depends on all of it, as before: with no javatools, with a javatools that
has no ModuleApi, for a file it cannot read as a module, or in a JVM too
old to load javatools, which the compile itself runs in a JVM of its own.

The API files have the extension .api: they are only compared, never
loaded, and are not valid module files.
@lagergren
lagergren force-pushed the lagergren/deterministic-xtc branch from de8e0d4 to 739d0f3 Compare November 18, 2026 11:24
@lagergren
lagergren marked this pull request as ready for review November 18, 2026 12:09
The compiler stamps every module with the epoch, so that the same source
compiles to the same bytes. To record a time instead, xtc build
--timestamp stamps the modules it compiles with the time of the compile
("now") or with an ISO-8601 instant, such as a release commit's time. The
Gradle plugin passes it from the compiler extension's timestamp setting,
which is unset by default. A module that is up to date is not compiled
again and keeps its stamp; --rebuild restamps it.

A module's description shows the timestamp only when a time was set,
instead of always showing the epoch, which says nothing.
@lagergren
lagergren requested a review from ggleyzer November 18, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants