Feature: Support saving scan result inside a gradle task (for JVM libs)

[F43nd1r]

Context: We're currently running ComposablePreviewScanner via Roborazzi plugin. This creates (on my PC) 16 shards, which each spent almost a minute scanning for previews to then throw away 15/16th of the result and run on their part. We have about ~800 previews.

Request: Support running at build time to be able to cache the scan result for shards (and potentially even between runs if sources did not change).

From my understanding this would require to use overrideClasspath in https://github.com/sergio-sastre/ComposablePreviewScanner/blob/master/core/src/main/java/sergio/sastre/composable/preview/scanner/core/scanner/ComposablePreviewScanner.kt#L30

We could then dump the found previews in a file, and each shard can read what it needs easily.

Alternative: If you feel like this is the wrong way to go about optimizing this, I'm happy to discuss 🙂

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• Feature: support saving scan result in assets inside gradle task (instrumentation libs) #38

🔗 Related PRs

#73 - Add support to easily use Classpath for ScreenshotTest Source Set in legacy AGP Versions [merged]
#98 - make AndroidComposablePreviewScanner usable in desktop [merged]
#106 - Release/0.8.1 [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[sergio-sastre]

Hi @F43nd1r ! Thanks for opening the issue.

That is already supported by this library. Take a look at core/src/main/java/sergio/sastre/composable/preview/scanner/core/scanresult/dump/ScanResultDumper.kt

It is also documented in the README.md, although I am aware it is not very intuitive. Check section:
Scanning Source Options (packages, files, inputStreams)
However, it is a bit trickier since as of today, you currently need an extra test to run the ScanResultDumper. I am planning to improve that in a future version, so it can be done via a Gradle task.

That is also why Takahiro and I also discussed about integrating this for Roborazzi when providing sharding support but honestly, it was much more work than it was worthy for the first version.

However, that each shard takes 1 minute to scan for Previews is very remarkable. Are you sure that it is the scanning what takes long or the test execution in general? Moreover, what is the time with only 1 shard?
Having so many threads also means you consume much more RAM, and since Robolectric already requires quite a lot, having 16 threads sounds far from optimal even if you have 32 GB RAM machine with 16 cores.

I recommend that you try out with much lower numbers (3-4) and see what values you get. More threads != faster execution.

[F43nd1r]

However, that each shard takes 1 minute to scan for Previews is very remarkable. Are you sure that it is the scanning what takes long or the test execution in general? Moreover, what is the time with only 1 shard?
Having so many threads also means you consume much more RAM, and since Robolectric already requires quite a lot, having 16 threads sounds far from optimal even if you have 32 GB RAM machine with 16 cores.

It looks like it's actually 30s - the 1 minute is what I was seeing included some gradle work, jvm forking and the fact that forks don't start perfectly parallel.
Also it seems the testParameters are called an additional time per fork but I haven't found out where that call comes from, possibly android studio?

Available Memory: 64GB. Cores: 16

Shards	Preview Scanning per Shard	Total Runtime	Used Memory (inkl. Android Studio)
16	30s	6:00m	20GB
8	27s	4:10m	17GB
4	22s	3:40m	15GB
2	18s	4:40m	14GB
1	17s	6:05m	12GB

Also experimented with shards without jvm forking but got worse results.

Another thing I found is: running in the same gradle daemon will reduce preview scanning time gradually, maybe some tiered compilation going on 🤔

However, it is a bit trickier since as of today, you currently need an extra test to run the ScanResultDumper. I am planning to improve that in a future version, so it can be done via a Gradle task.

I guess being able to do it via gradle task is what I was requesting.

---

Looks like we'll be limiting to 4 shards, but I guess doing the scan in a gradle task would still be a valid improvement.
Thanks for the suggestion!

[sergio-sastre]

@F43nd1r
Glad it helped!

Let me help clarify other questions :)

Also it seems the testParameters are called an additional time per fork but I haven't found out where that call comes from, possibly android studio?

This is an issue in Robolectric, and has to do with the Parameterized Test Runners. I opened the issue here
robolectric/robolectric#10605
I am sure it is related to Robolectric, because that behaviour does not happen with Paparazzi, and in also Robolectric-but-not-Roborazzi tests it is still reproducible

I guess being able to do it via gradle task is what I was requesting.

It is actually closely related to this issue: #38
However, I have realised that what was complicated, was to dump the files in assets, which is necessary for instrumentation tests.
Dumping in local files in a gradle task should be possible. I'll give it a try and add the corresponding documentation with an example if that is the case, but this could take some time since I am busy with other content currently.

Available Memory: 64GB. Cores: 16

Shards	Preview Scanning per Shard	Total Runtime	Used Memory (inkl. Android Studio)
16	30s	6:00m	20GB
8	27s	4:10m	17GB
4	22s	3:40m	15GB
2	18s	4:40m	14GB
1	17s	6:05m	12GB

The Preview Scanning per Shard times look high though... Are you sure it is only due to the "scanning"? For me it looks more like all the tasks that are executed (e.g. Robolectric SDK download) before the tests.
ComposablePreviewScanner provides a method in its api to print ONLY the scanning times in the console:

AndroidComposablePreviewScanner()
    // Optional to log scanning info like scanning time or amount of previews found
    .enableScanningLogs()
    ...
    .getPreviews()

I am not sure if you've used that to measure those times. Could you enable it and check how long it actually takes to scan Previews in your project (you'll see it executes 2 times per shard, which is a Robolectric issue as I explained above)?
Moreover, are you using .scanPackageTrees() or scanAllPackages()? the latter takes definitely longer and might search for previews in unnecessary packages.

That info would be very helpful :)
Thanks in advance!

[F43nd1r]

I tested with

timedValue { AndroidComposePreviewTester().testParameters() }

Using scanPackageTrees on our root package. There are quite a few classes, but as our previews are spread around many subpackages there wasn't an easy way to slim it down.

[F43nd1r]

Unfortunately the roborazzi generated code doesn't allow me to turn on .enableScanningLogs() easily, and it's using some internal stuff so just copying it didn't work either.
But I did collect some more stats: We have about 1k files with about 100k LoC in the main source set.

[sergio-sastre]

@F43nd1r
Oh, interesting. Thanks for trying it out anyway.

So let’s proceed as follows.

Let me test if save the scanning in a file inside a Gradle task is possible and document it in the README in case it works.

I am not sure how much faster it will execute the tests though, but hope to see some small wins at least.

I’ll let you know in this issue once I’ve tested it, but estimate that it might take me a couple of weeks.
I hope it is not inconvenient.

I’ll label this as “documentation improvement” in the meanwhile

[sergio-sastre]

@F43nd1r Sorry, I've not found the time to test it properly yet. I hope I will find time during the upcoming week

[F43nd1r]

No worries, thanks for the update

[sergio-sastre]

@F43nd1r
It turns out it has similar problems to #38.
Basically, it works, but when used in a Gradle task, it scans only pure @Previews, so @PreviewFontSizes and similar or custom @Previews built on top of @Preview are not detected.

I think I will mark this as a feature-request together with #38 and expect to release it in 0.10.0 (in 2 versions). Release 0.9.0 is likely coming this next week.