Everything else in this repository tests the runner as source. An operator runs a
downloaded file, and the distance between those two is where release defects live:
a build flag that only applies to one platform, an archive missing a document, a
binary that needs a library the build machine happened to have.
Add a job that closes that distance. It downloads the published artefact for each
platform, verifies its checksum against the published file, runs it against a fresh
clone of the default branch, and asserts the result.
Two properties make it worth having rather than ceremonial.
It uses the published artefact, not one rebuilt in the job. A rebuild tests the
build a second time and says nothing about what was published, which is the thing
an operator will actually download.
It asserts on the output rather than only on the exit code. Exit zero with no output
at all is what a binary that failed to find the tree also produces, and that is
precisely the release defect this job exists to catch.
Run it after publishing and on a schedule afterwards, because an artefact that
becomes unfetchable weeks later is a real failure that nothing else here notices.
Done when the job exists, downloads and verifies each published artefact, runs it
against a fresh clone, asserts on both output and exit code, and has passed against
a real release.
Everything else in this repository tests the runner as source. An operator runs a
downloaded file, and the distance between those two is where release defects live:
a build flag that only applies to one platform, an archive missing a document, a
binary that needs a library the build machine happened to have.
Add a job that closes that distance. It downloads the published artefact for each
platform, verifies its checksum against the published file, runs it against a fresh
clone of the default branch, and asserts the result.
Two properties make it worth having rather than ceremonial.
It uses the published artefact, not one rebuilt in the job. A rebuild tests the
build a second time and says nothing about what was published, which is the thing
an operator will actually download.
It asserts on the output rather than only on the exit code. Exit zero with no output
at all is what a binary that failed to find the tree also produces, and that is
precisely the release defect this job exists to catch.
Run it after publishing and on a schedule afterwards, because an artefact that
becomes unfetchable weeks later is a real failure that nothing else here notices.
Done when the job exists, downloads and verifies each published artefact, runs it
against a fresh clone, asserts on both output and exit code, and has passed against
a real release.