Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,57 @@ For example, after adding `api "org.slf4j:slf4j-api:${versions.slf4j}"` to [plug

Ensure that `./gradlew :plugins:discovery-ec2:check` passes before submitting changes.

### Dependency Verification

Every artifact the build downloads, including the dependencies of the build itself (`buildSrc` / `build-tools`) and
the Gradle plugins applied by the build scripts, is checksum verified against
[gradle/verification-metadata.xml](gradle/verification-metadata.xml). Gradle performs this check before an artifact is
placed on any classpath, so a tampered dependency fails the build before its code can run. This protects the build
against a compromised upstream artifact, which is a gap that the per-project `licenses/*.jar.sha1` files do not cover
because they are only checked by the `dependencyLicenses` task, and only for shipped dependencies.

Adding, removing or upgrading a dependency changes the set of artifacts that gets resolved, so the metadata has to be
regenerated. For a single dependency change, the quick way is to fetch the new artifact and then rewrite the metadata
from the local cache:

```
./gradlew <task that resolves the changed dependency> -Dorg.gradle.dependency.verification=off
./gradlew --write-verification-metadata sha256 <same task> --offline
```
Comment on lines +402 to +404

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This two-step process to update dependencies can save a lot of time, since the default "online" mode for --write-verification-metadata seems to read from the various artifactories for every single task (as I understand it).

The :qa:os subprojects iterate through every listed OS and every version of OpenSearch defined in Version.java and fetches Jackson + JUnit dependencies. On my laptop, the online mode takes over 6 minutes (though I've seen it go as high as 9 minutes), while the offline mode is closer to 1 minute.

Removing the :qa:os subprojects and removing most of the unused constants in Version.java cuts the online metadata regeneration time to 1m30s, which is much closer to our old updateSHAs task.

The two-step process exists as a workaround until then, but it didn't work for me with -Dsandbox-enabled=true, because it insisted that arrow-compression-18.1.0.jar was not cached for offline verification (though I could clearly see it in my Gradle cache).

So, to summarize:

  1. If you're updating dependencies, not in sandbox, this two step process works and is fast.
  2. If you're updating sandbox dependencies, you need to do the online regeneration with something like ./gradlew --write-verification-metadata sha256 precommit -Denable.sandbox=true. Go grab a coffee.
  3. If we remove the :qa:os tasks and remove some cruft from Version.java, then this whole section can be a lot shorter and just document the online regeneration step (since it will be decently fast).


The first command populates the Gradle cache, since verification would otherwise reject the not-yet-recorded artifact.
The second records its checksum. Regeneration only ever adds entries, so a narrow task is enough: nothing that was
already recorded gets dropped. Passing `--offline` matters a great deal, because without it Gradle re-checks every
recorded component against the remote repositories, which takes minutes rather than seconds. If an artifact is not in
the cache yet, the offline run fails naming it rather than silently omitting it.

To regenerate the file from scratch, drop it and resolve everything, which needs network access and takes several
minutes:

```
./gradlew --write-verification-metadata sha256 resolveAllDependencies precommit --continue
```

Both task names matter. `resolveAllDependencies` resolves every resolvable configuration, while `precommit` additionally
triggers the tools that resolve their own dependencies into detached configurations when they run, such as Spotless
pulling in `google-java-format`. Generating against only one of them produces metadata that is missing entries, and the
build then fails with `Dependency verification failed` naming the artifacts that were not recorded. Some
`resolveAllDependencies` tasks in `qa` and `distribution` fail unless the backwards compatibility distributions have
been built locally; `--continue` lets the rest of the run proceed, and those distributions are covered by a `<trust>`
rule rather than a checksum anyway.

Because regeneration only adds entries, the file also never loses the record of a dependency that has been removed.
Deleting the file and regenerating it in full is the only way to prune those.

Review the resulting diff before committing it. Regeneration records whatever is downloaded at that moment, so entries
should be added because a dependency genuinely changed, not because a checksum drifted underneath an unchanged
dependency. A changed checksum for a dependency whose version did not change is exactly the situation this check exists
to surface.

OpenSearch's own distributions, which `DistributionDownloadPlugin` downloads for backwards compatibility and packaging
tests, are covered by a `<trust>` rule rather than by a checksum. Snapshot distributions are rebuilt continuously, so
there is no stable checksum to record. Third-party dependencies must never be added to that list.

### Editor / IDE Support

IntelliJ IDEs can [import](https://blog.jetbrains.com/idea/2014/01/intellij-idea-13-importing-code-formatter-settings-from-eclipse/) the [settings file](buildSrc/formatterConfig.xml), and / or use the [Eclipse Code Formatter](https://plugins.jetbrains.com/plugin/6546-eclipse-code-formatter)
Expand Down
4,257 changes: 4,257 additions & 0 deletions gradle/verification-metadata.xml

Large diffs are not rendered by default.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
13a748ea3e329fa220076e021b45c8391b32420c

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
921bd2092b0c539b2876de7063d55c72edcd05d3

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0b900dd7125fa16cfdf46135d3ffb3243d0f8b88

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
a4f075bf4cc1ee814ab98d69c1612786c2f42bc3
Loading