Knapsack Jest runs not respecting cache

[what-is-a-crow]

We've been using Knapsack Pro with our Jest tests for several months now with no problems; however, when we bump our version of Jest to the latest (29.7.0), the runs in Github Actions go from ~10 minutes each to ~50. The initial load of the app has always been very slow; however, it's a one-time cost, as I believe Jest uses a local cache for subsequent runs. Instead of something like this:

// ...Knapsack API response...
PASS a.test.ts (3 s)
PASS b.test.ts (2 s)
PASS c.test.ts (3 s)
// ...Knapsack API response...
PASS d.test.ts (3 s)
PASS e.test.ts (2 s)
PASS f.test.ts (3 s)

...we're seeing this:

// ...Knapsack API response...
PASS a.test.ts (75 s)
PASS b.test.ts (2 s)
PASS c.test.ts (3 s)
// ...Knapsack API response...
PASS d.test.ts (74 s)
PASS e.test.ts (2 s)
PASS f.test.ts (3 s)

...with the first test of each Jest run taking longer than a minute.

Is our suspicion correct, that Jest is for some reason ignoring the cache and reloading all the modules each time now?

[ArturT]

Please share the knapsack pro command that you use to run tests. Do you pass any options to Jest?

[what-is-a-crow]

The command looks like this: knapsack-pro-jest --runInBand --ci --config=jest.config.js --coverage --silent. The Knapsack options are:

          KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_LOG_LEVEL: info
          KNAPSACK_PRO_TEST_FILE_PATTERN: "**/__tests__/**/*.+(spec|test).[jt]s?(x)"
          KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN: "{**/cypress/**}"
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_COVERAGE_DIRECTORY: coverage
          JEST_JUNIT_UNIQUE_OUTPUT_NAME: true
          NODE_OPTIONS: --max-old-space-size=14436

[ArturT]

What version of Jest have you used before?
What is your CI provider?
Do you use Create React App?
Did you update other things than Jest?
Have you changed any app configuration other than Jest that could impact the performance?

Please share a link to the slow CI build in the Knapsack Pro user dashboard. You can also share a link to a CI build that is working fine (before Jest update).

Please share an output from a CI for the slow CI build. You can send the private data to the support email. Thank you.

[what-is-a-crow]

We're coming from Jest version 27.5.1.
CI provider is Github Actions.
We do not use Create React App.
A good handful of other dependencies got updated.

Dependencies that got bumped to support the Jest upgrade:

@types/jest
@typescript-eslint/eslint-plugin
@typescript-eslint/parser
babel-jest
jest-environment-jsdom (new)
jest-jasmine2 (new)
ts-jest
typescript

Dependencies that got bumped while trying to figure out the performance issue:
@knapsack-pro/jest itself to 8.0.1

@babel/core
@babel/plugin-proposal-export-default-from
@babel/plugin-transform-runtime
@babel/preset-env
@babel/preset-react
@babel/runtime

I've tweaked app configuration in a few places but mostly just required changes to support the Jest upgrade; nothing that seems like it might impact this.

Here is a build experiencing slowness
Here is a build that is working fine.

I'll dig up some CI output and send it along.

Thank you for your help so far, @ArturT !

[3v0k4]

Sorry for the troubles, @what-is-a-crow

I tried with several configurations, but couldn't reproduce the issue.

I'm surprised because Knapsack Pro should respect the jest cache.

It would be fantastic if you could share a minimal repository that reproduces the problem. That would be the quickest way for me to investigate further.

If that's not an option, I'd ask you to please help me with the following (feel free to send an email if anything is too sensitive to paste here):

1. Share your jest.config.js; I'll see if it helps me reproduce.

2. Share your package.json (and package-lock.json); I'll see if it helps me reproduce.

3. Run npx jest --version and npx jest --show-config (both 27.5.1 and 29.7.0) on CI and paste the output here; this will show the entire jest configuration.

4. Run on CI jest 29.7.0 (npx jest ...) twice without Knapsack Pro on a couple of files and see if the second run is cached or not (i.e., the penalty occurs only on the first run); this will ensure that jest uses the cache when run directly.

5. Run on CI jest 27.5.1 with Knapsack Pro using the --no-cache flag and see if it reproduces the same issue; this will show us if we can reproduce the same behavior in the setup that currently works for you.

6. Create a file setup.js that contains:

   const fs = require('node:fs')
   const cp = require('node:child_process');
   
   beforeAll(() => {
     const jestConfig = JSON.parse(cp.execSync('npx jest --show-config'))
   
     jestConfig.configs.forEach((config, i) => {
       console.log({
         configIndex: i,
         cache: config.cache,
         cacheDirectory: config.cacheDirectory,
       })
     })
   })
   
   afterEach(() => {
     const jestConfig = JSON.parse(cp.execSync('npx jest --show-config'))
   
     jestConfig.configs.forEach((config, i) => {
       const tree = (dirs, depth) => {
         let acc = ''
         if (dirs.length == 0) return acc;
   
         dirs.forEach(dir => {
           const dir_ = depth === 0 ? dir : dir.split('/')[dir.split('/').length-1]
           const spacer1 = Array(depth * 4).fill(' ').join('')
           acc = acc + `\n${spacer1}${dir_}`
           const dirents = fs.readdirSync(dir, { withFileTypes: true })
           const spacer2 = Array((depth+1) * 4).fill(' ').join('')
           dirents.filter(dirent => dirent.isFile()).forEach(dirent => acc += `\n${spacer2}${dirent.name}`)
           const innerDirs = dirents.filter(dirent => dirent.isDirectory()).map(dirent => dirent.path+'/'+dirent.name)
           acc = acc + tree(innerDirs, depth+1)
         })
         return acc
       }
   
       console.log(tree([config.cacheDirectory], 0))
     })
   })

   Add to your jest config the following:

   "setupFilesAfterEnv": [
     "<rootDir>/setup.js"
   ]
   

   Run on CI jest (both 27.5.1 and 29.7.0) with Knapsack Pro (make sure --runInBand is used for this) and share the output; this will show us what Jest is caching. You may want to limit the tests run with KNAPSACK_PRO_TEST_FILE_PATTERN.

Thanks 🙏

[what-is-a-crow]

@3v0k4 thanks so much for your attention here. I set up three separate experiment PRs and gathered some data; I've included the findings and the artifacts you requested in an email to @ArturT / support.