diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 93fb6470cfa15..e27ceb7e76155 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -385,6 +385,57 @@ For example, after adding `api "org.slf4j:slf4j-api:${versions.slf4j}"` to [plug Ensure that `./gradlew :plugins:discovery-ec2:check` passes before submitting changes. +### Dependency Verification + +Every artifact the build downloads, including the dependencies of the build itself (`buildSrc` / `build-tools`) and +the Gradle plugins applied by the build scripts, is checksum verified against +[gradle/verification-metadata.xml](gradle/verification-metadata.xml). Gradle performs this check before an artifact is +placed on any classpath, so a tampered dependency fails the build before its code can run. This protects the build +against a compromised upstream artifact, which is a gap that the per-project `licenses/*.jar.sha1` files do not cover +because they are only checked by the `dependencyLicenses` task, and only for shipped dependencies. + +Adding, removing or upgrading a dependency changes the set of artifacts that gets resolved, so the metadata has to be +regenerated. For a single dependency change, the quick way is to fetch the new artifact and then rewrite the metadata +from the local cache: + +``` +./gradlew -Dorg.gradle.dependency.verification=off +./gradlew --write-verification-metadata sha256 --offline +``` + +The first command populates the Gradle cache, since verification would otherwise reject the not-yet-recorded artifact. +The second records its checksum. Regeneration only ever adds entries, so a narrow task is enough: nothing that was +already recorded gets dropped. Passing `--offline` matters a great deal, because without it Gradle re-checks every +recorded component against the remote repositories, which takes minutes rather than seconds. If an artifact is not in +the cache yet, the offline run fails naming it rather than silently omitting it. + +To regenerate the file from scratch, drop it and resolve everything, which needs network access and takes several +minutes: + +``` +./gradlew --write-verification-metadata sha256 resolveAllDependencies precommit --continue +``` + +Both task names matter. `resolveAllDependencies` resolves every resolvable configuration, while `precommit` additionally +triggers the tools that resolve their own dependencies into detached configurations when they run, such as Spotless +pulling in `google-java-format`. Generating against only one of them produces metadata that is missing entries, and the +build then fails with `Dependency verification failed` naming the artifacts that were not recorded. Some +`resolveAllDependencies` tasks in `qa` and `distribution` fail unless the backwards compatibility distributions have +been built locally; `--continue` lets the rest of the run proceed, and those distributions are covered by a `` +rule rather than a checksum anyway. + +Because regeneration only adds entries, the file also never loses the record of a dependency that has been removed. +Deleting the file and regenerating it in full is the only way to prune those. + +Review the resulting diff before committing it. Regeneration records whatever is downloaded at that moment, so entries +should be added because a dependency genuinely changed, not because a checksum drifted underneath an unchanged +dependency. A changed checksum for a dependency whose version did not change is exactly the situation this check exists +to surface. + +OpenSearch's own distributions, which `DistributionDownloadPlugin` downloads for backwards compatibility and packaging +tests, are covered by a `` rule rather than by a checksum. Snapshot distributions are rebuilt continuously, so +there is no stable checksum to record. Third-party dependencies must never be added to that list. + ### Editor / IDE Support IntelliJ IDEs can [import](https://blog.jetbrains.com/idea/2014/01/intellij-idea-13-importing-code-formatter-settings-from-eclipse/) the [settings file](buildSrc/formatterConfig.xml), and / or use the [Eclipse Code Formatter](https://plugins.jetbrains.com/plugin/6546-eclipse-code-formatter) diff --git a/gradle/verification-metadata.xml b/gradle/verification-metadata.xml new file mode 100644 index 0000000000000..2c538a282fbf4 --- /dev/null +++ b/gradle/verification-metadata.xml @@ -0,0 +1,4257 @@ + + + + false + false + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/sandbox/libs/analytics-framework/licenses/jackson-core-2.22.1.jar.sha1 b/sandbox/libs/analytics-framework/licenses/jackson-core-2.22.1.jar.sha1 deleted file mode 100644 index e2c41405d169c..0000000000000 --- a/sandbox/libs/analytics-framework/licenses/jackson-core-2.22.1.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -da7ffb60088d7e8f37ecdd3b617520971cc7b9bf \ No newline at end of file diff --git a/sandbox/libs/analytics-framework/licenses/jackson-core-2.22.2.jar.sha1 b/sandbox/libs/analytics-framework/licenses/jackson-core-2.22.2.jar.sha1 new file mode 100644 index 0000000000000..67dfe67ab4b45 --- /dev/null +++ b/sandbox/libs/analytics-framework/licenses/jackson-core-2.22.2.jar.sha1 @@ -0,0 +1 @@ +13a748ea3e329fa220076e021b45c8391b32420c \ No newline at end of file diff --git a/sandbox/libs/analytics-framework/licenses/jackson-databind-2.22.1.jar.sha1 b/sandbox/libs/analytics-framework/licenses/jackson-databind-2.22.1.jar.sha1 deleted file mode 100644 index 7b38d1b99d9d0..0000000000000 --- a/sandbox/libs/analytics-framework/licenses/jackson-databind-2.22.1.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -9e2fb91831cce9cb9262909cd76647508949f232 \ No newline at end of file diff --git a/sandbox/libs/analytics-framework/licenses/jackson-databind-2.22.2.jar.sha1 b/sandbox/libs/analytics-framework/licenses/jackson-databind-2.22.2.jar.sha1 new file mode 100644 index 0000000000000..de250cf9085be --- /dev/null +++ b/sandbox/libs/analytics-framework/licenses/jackson-databind-2.22.2.jar.sha1 @@ -0,0 +1 @@ +921bd2092b0c539b2876de7063d55c72edcd05d3 \ No newline at end of file diff --git a/sandbox/plugins/analytics-backend-datafusion/licenses/jackson-datatype-jdk8-2.22.1.jar.sha1 b/sandbox/plugins/analytics-backend-datafusion/licenses/jackson-datatype-jdk8-2.22.1.jar.sha1 deleted file mode 100644 index 8aec5b1d3886e..0000000000000 --- a/sandbox/plugins/analytics-backend-datafusion/licenses/jackson-datatype-jdk8-2.22.1.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -ef21a94e7b5603f57ca1efe9215513d210d96320 \ No newline at end of file diff --git a/sandbox/plugins/analytics-backend-datafusion/licenses/jackson-datatype-jdk8-2.22.2.jar.sha1 b/sandbox/plugins/analytics-backend-datafusion/licenses/jackson-datatype-jdk8-2.22.2.jar.sha1 new file mode 100644 index 0000000000000..ef40d77269bbb --- /dev/null +++ b/sandbox/plugins/analytics-backend-datafusion/licenses/jackson-datatype-jdk8-2.22.2.jar.sha1 @@ -0,0 +1 @@ +0b900dd7125fa16cfdf46135d3ffb3243d0f8b88 \ No newline at end of file diff --git a/sandbox/plugins/analytics-engine/licenses/jackson-dataformat-yaml-2.22.1.jar.sha1 b/sandbox/plugins/analytics-engine/licenses/jackson-dataformat-yaml-2.22.1.jar.sha1 deleted file mode 100644 index 597173dcd7fc3..0000000000000 --- a/sandbox/plugins/analytics-engine/licenses/jackson-dataformat-yaml-2.22.1.jar.sha1 +++ /dev/null @@ -1 +0,0 @@ -af5fde2414e4a8d2617e7890f7068c26b5b66a33 \ No newline at end of file diff --git a/sandbox/plugins/analytics-engine/licenses/jackson-dataformat-yaml-2.22.2.jar.sha1 b/sandbox/plugins/analytics-engine/licenses/jackson-dataformat-yaml-2.22.2.jar.sha1 new file mode 100644 index 0000000000000..78dcb210d0996 --- /dev/null +++ b/sandbox/plugins/analytics-engine/licenses/jackson-dataformat-yaml-2.22.2.jar.sha1 @@ -0,0 +1 @@ +a4f075bf4cc1ee814ab98d69c1612786c2f42bc3 \ No newline at end of file