Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions docs/coverage-map-model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Coverage map semantic model

The semantic model lives under `com.sap.oss.smarttestpicker.coverage.model`; JSON construction and
parsing live separately under `coverage.serialization`. Model records defensively copy maps, lists,
and sets, expose no setters, and use value types for revisions, test identities, containers, and
shards. This separation lets selection and future builders depend on meaning rather than dictionary
indexes or JSON layout.

The main types are `CoverageMap`, `CoverageMapRevision`, `TestIdentity`, `TestCoverage`,
`TestContainer`, `SetupScope`, `UnmappedTest`, `TestInventory`, `CollectionExpectation`,
`CollectionSummary`, `Completeness`, `TestOutcome`,
`GeneratorProvenance`, `MapStatistics`, `CoverageFragment`, and `MethodCoverageReference`.

## Setup is first-class

Static initialization, `@BeforeAll`, inherited setup, nested setup, and cached framework contexts do
not necessarily belong to one test class. `SetupScope` therefore maps covered classes to a set of
affected containers. This lets backlog 5b conservatively select every logical test in every affected
container without schema changes.

## Failed and unmapped tests

Test outcome and coverage collection status are independent. A failed test with a valid coverage
collection remains a mapped test with `outcome = FAIL`; its coverage is not discarded. A test is an
`UnmappedTest` only when no valid mapping exists, including a collection failure. Backlog 5b owns
the future selection policy for these states.

`COLLECTED_EMPTY` is different: the test and collector completed successfully and observed no
production coverage. `COLLECTION_FAILED` belongs in `unmapped`, never in a successful test mapping.

## Independent inventory and completeness

`Completeness.from` requires both an independent `CollectionExpectation` and a collector-side
`CollectionSummary`. The expectation combines `TestInventory` with expected shard identities;
neither expected tests nor expected shards can be inferred from collector output. Inventory discovery belongs to 5c/5d. Completeness uses identity
set differences and separately records missing, unexpected and duplicate tests and missing or
duplicate shards. Equal counts with different identities are incomplete.

Dynamic sharding has a global expected test set and an expected shard set; there is deliberately no
shard-to-test assignment in the model.

## Lifecycle

`CoverageMapLifecycleState` represents `FRAGMENT`, `CANDIDATE`, and `PUBLISHED`. Only a validated
`PUBLISHED` map may be selector input. This task defines the distinction; atomic publication,
retention, and storage belong to backlog 5d.

## Deliberate boundaries

This contract does not implement fragment merging or a map builder (5c/5d), runtime coverage and
test discovery (5c), setup-scope or unmapped-test selection policy (5b), storage/publication (5d),
Jenkins orchestration (5d), or arbitrary base/head revision handling (5e). It also does not alter
Gradle or Maven adapters.

## Version impact

Schema v1 is not compatible with legacy maps. Legacy test keys use `SimpleClass#method_7hexHash`,
from which the FQN cannot be reconstructed, so maps must be regenerated. The semantic selector API
need not change in Task 5a and the existing CLI output schema remains unchanged; future v1 reader
adoption will be an explicit integration step. Because Core and CLI artifact content changes, the
next release must not reuse `0.1.0`; the recommended next artifact version is `0.2.0`.

The RAD 1 evaluation repository is frozen at commit `a53e5c7` and is not modified or regenerated.
167 changes: 167 additions & 0 deletions docs/coverage-map-schema.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
# Coverage map schema v1

Schema v1 extends STP's existing indexed JSON format. It is the first frozen coverage-map
contract; legacy maps without `schemaVersion` remain readable only through the legacy reader
and cannot be converted because their hashed simple-name test identities do not contain the
original fully-qualified class name.

`CoverageMapContract.SCHEMA_VERSION` in STP Core is the single authority for map and fragment
validation. Compatibility requires exact equality. A higher or lower integer is rejected;
integer ordering does not imply compatibility. `ExecutionPlanContract.VERSION` similarly owns
execution-plan version 1 without changing the execution-plan wire format.

## Indexed physical form

The first implementation writes class and method coverage in one artifact:

```json
{
"schemaVersion": 1,
"revision": "abc123",
"generatedAt": "2026-08-30T10:00:00Z",
"generator": {
"stpVersion": "0.2.0",
"coverageRuntimeVersion": "0.1.0",
"jacocoVersion": "0.8.15",
"jdkVersion": "17"
},
"classIndex": ["com.foo.ConfigService", "com.foo.OrderService"],
"methodIndex": ["com.foo.OrderService#placeOrder"],
"containerIndex": ["com.foo.BarTest", "com.foo.FooTest"],
"testIndex": ["com.foo.BarTest#slow", "com.foo.FooTest#a"],
"tests": [
{"test": 1, "classes": [1], "methods": [0], "outcome": "FAIL", "collectionStatus": "COLLECTED_WITH_COVERAGE"}
],
"unmapped": [
{"test": 0, "reason": "TIMEOUT"}
],
"setupScopes": [
{"id": "scope-1", "type": "CONTAINER", "coveredClasses": [0], "affectedContainers": [1]}
],
"completeness": {
"expectedTests": [0, 1],
"reportedTests": [0, 1],
"expectedShards": ["01"],
"completedShards": ["01"],
"missingTests": [],
"unexpectedTests": [],
"missingShards": [],
"duplicateShards": [],
"duplicateTests": []
},
"statistics": {
"expectedTests": 2,
"mappedTests": 1,
"unmappedTests": 1,
"setupScopes": 1,
"classEdges": 1,
"methodEdges": 1
},
"lifecycleState": "PUBLISHED",
"checksum": "sha256:<64-lowercase-hex-digits>"
}
```

Every interning table and every reference collection is sorted. Test identities are interned
because their FQNs otherwise recur in mappings, unmapped entries, and both completeness sets.
`classMetrics` is intentionally excluded: it is reporting data and is not read by selection.

## Test identity

The logical identity is `fully.qualified.BinaryClass#declaredMethod`. Nested classes use JVM
binary naming, for example `com.foo.OuterTest$Inner#nestedTest`, matching the names accepted by
build-tool filters. All invocations of one declared method collapse into one identity:

* `@ParameterizedTest` and `@RepeatedTest` invocations are combined;
* `@TestTemplate` invocations are combined under the template method;
* `@TestFactory` and its dynamic tests are combined under the factory method.

Whether a runtime can always recover the declaring factory method when a dynamic test lacks a
usable `MethodSource` remains an open 5c question. Schema v1 does not introduce invocation IDs.

## Setup scopes

A setup scope independently relates covered production classes to one or more affected test
containers. This represents nested, inherited, framework, and shared-context setup without
assigning coverage to whichever test happened to initialize it first. Supported enum values are
`CONTAINER`, `NESTED_CONTAINER`, `INHERITED_SETUP`, `SHARED_CONTEXT`, and `FRAMEWORK_SETUP`.
Only `CONTAINER` is currently expected to be producible. The other values are schema placeholders,
not claims of collector capability. Covered classes and containers reference shared tables.

## Test outcome, unmapped tests, and collection status

`TestOutcome` (`PASS` or `FAIL`) is independent of `CollectionStatus`. Both passing and failing tests
may have `COLLECTED_WITH_COVERAGE` or `COLLECTED_EMPTY`; either is a mapped, reported test for
completeness. An unmapped test has no `tests` entry because no valid coverage mapping exists. A
collection failure remains `UnmappedReason.COLLECTION_FAILED` and is never synthesized as successful
empty coverage. Other unmapped reasons diagnose why a valid mapping is unavailable; `FAILED` does
not imply that every failed test is unmapped. Selection semantics remain a 5b concern.

## Completeness and shards

Completeness compares global expected and reported identity sets, never counts. Reported means
mapped plus unmapped. `CollectionExpectation` independently owns expected shard identities;
`CollectionSummary` contains only completed shard reports and other collected facts. Sharding is
dynamic and there is no per-shard expected-test assignment. Missing,
unexpected and duplicate identities and missing or duplicate shards prevent publication.

Generator versions and JDK/JaCoCo values are diagnostic and do not reject a map. Statistics are
also descriptive; completeness is never inferred from their counts.

A `CANDIDATE` validation fixture may use JSON `null` for `completeness` when its source has no
independent inventory. Such a map can be codec-round-tripped, but validation still reports
`LIFECYCLE_NOT_PUBLISHED`, and it is not selector input. An absent inventory must never be replaced
with `expectedTests = collectedTests`.

## Determinism and integrity

The checksum is SHA-256 over compact canonical JSON with the `checksum` member omitted. The final
wire document adds that checksum after hashing. All identities, dictionaries, references, scope
members, shards, and entries are sorted before serialization.

> Coverage map SHA-256 provides integrity and corruption detection. It is not an authenticity or trust mechanism.

Trust in a map must come from its publication/distribution boundary; the checksum only identifies
and verifies the exact bytes inside that boundary.

## Reserved separable method coverage

Schema v1 reserves an optional checksum-covered descriptor:

```json
"methodCoverage": {
"artifact": "coverage-methods.json.gz",
"sha256": "sha256:<64-lowercase-hex-digits>"
}
```

The first implementation does not use it and writes methods inline. A later implementation may
publish an immutable side artifact for lazy loading without changing the base schema. Resolution,
publication, and fallback behavior belong to 5b/5d. A declared but missing or corrupt side artifact
is not equivalent to a map that deliberately contains no method descriptor.

## Provisional fragment contract

```json
{
"schemaVersion": 1,
"revision": "abc123",
"shardId": "07",
"tests": {
"com.foo.FooTest#a": {
"classes": ["com.foo.OrderService"],
"methods": ["com.foo.OrderService#placeOrder"],
"outcome": "PASS",
"collectionStatus": "COLLECTED_WITH_COVERAGE"
}
},
"unmapped": [{"test": "com.foo.BarTest#b", "reason": "COLLECTION_FAILED"}],
"setupScopes": [],
"collection": {"completed": false}
}
```

Fragments from different revisions are not combinable. A fragment declares its shard but never
the expected shard set. `completed: false` differs from an absent fragment. This contract is
**PROVISIONAL** until 5c confirms how shard IDs enter the test JVM and whether the runtime can
reliably distinguish `COLLECTED_EMPTY` from `COLLECTION_FAILED`.
78 changes: 78 additions & 0 deletions docs/jgrapht-schema-v1-validation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# JGraphT coverage-map schema-v1 validation

## Dataset and conversion

The validation reads, but does not modify or regenerate,
`../smart-test-picker-evaluation/jgrapht/results/test-coverage-map.json.gz`. The artifact was added at
evaluation-repository commit `df8f3785075773c96ce5e8baaa6e10d271b4b32d` and identifies JGraphT
revision `093b0c5ea006ba5b1d8b7a0212676bf8850cac6b`. Its physical format is gzip-compressed legacy JSON
with `classMetrics`, `metadata`, and 2,308 `testMappings`; mappings contain both class and method
coverage.

The validation helper removes the legacy seven-hex-character suffix and reconstructs the FQN from
JGraphT test sources. If a simple class name is ambiguous, an archived JUnit identity in the
evaluation PIT XML must disambiguate it. It never guesses an FQN. This converted 2,298 mappings
(99.57%); ten mappings were excluded and reported because their `ReversedDoublyLinkedListViewTest`
source class was unavailable. The two `IncomingOutgoingEdgesTest` mappings are safely distinguished
where archived JUnit evidence exists; one without such evidence remains excluded.

The semantic fixture is a `CANDIDATE`, has `completeness = null`, and uses an empty setup-scope list.
The null means no independent expected inventory is available. The empty scope list means setup
information is unavailable, not that JGraphT has no setup coverage. The fixture is consequently not
valid selector input and cannot be mistaken for a published map.

## Results

The complete legacy artifact has 565 unique covered classes, 3,151 unique covered methods, 60,556
test-to-class edges, and 213,049 test-to-method edges. After removing the ten unresolved mappings,
the converted source subset and the schema-v1 round trip both have exactly:

| Measure | Converted source | Round trip |
| --- | ---: | ---: |
| Tests | 2,298 | 2,298 |
| Unique classes | 561 | 561 |
| Unique methods | 3,099 | 3,099 |
| Test-to-class edges | 60,477 | 60,477 |
| Test-to-method edges | 212,656 | 212,656 |

All reconstructed `TestIdentity` values and semantic edge sets compare equal. Every test, class, and
method index resolves in range; dictionaries are sorted and contain no semantic duplicates. Nested
binary names are handled by preserving the `$` suffix after resolving the outer test class.

One representative run produced 1,920,176 raw bytes and 111,800 gzip bytes. Dictionary JSON members
occupied 412,691 bytes and the indexed `tests` member occupied 1,501,089 bytes (member sizes include
their JSON field framing and are diagnostic, not an additive file decomposition). The result is
broadly consistent with the earlier indexed/gzip expectation: dictionary indexing plus gzip reduces
this high-edge-density payload to about 6% of its raw schema-v1 size. The checksum was
`sha256:5561b8545399aff95e33dec02b05d9fb819191674019a9183e93adfe33168a2e` for that deterministic
fixture. Two serializations and serialize-deserialize-serialize were byte-identical.

Serializing the same semantic map with method sets removed measured the method-coverage contribution
at 1,246,745 raw bytes and 70,265 gzip bytes. This comparison is diagnostic and is not a proposal to
change the inline schema.

Representative in-process timings, averaged where practical, were approximately 217 ms semantic
construction, 79 ms serialization, 101 ms deserialization, and 241 ms semantic/dictionary
validation. These are diagnostics rather than thresholds. Heap delta/peak was not measured because
the Gradle test JVM does not provide a useful isolated peak without adding profiling infrastructure.

Corrupting the real-data-sized serialized document is rejected as `CHECKSUM_MISMATCH`. Changing its
schema version to 2 is rejected before v1 parsing as `INCOMPATIBLE_SCHEMA/HIGHER_SCHEMA_VERSION`.

## Running the validation

`JGraphTCoverageMapV1IntegrationTest` uses the sibling evaluation and JGraphT trees when present and
is skipped otherwise, so the normal test suite remains portable. Alternate read-only locations can
be supplied with JVM properties `stp.jgraphtCoverageMap` and `stp.jgraphtSourceRoot`. It neither
executes JGraphT tests nor invokes a coverage collector.

## Limitations

- No new coverage was collected, and the full JGraphT test suite was not run.
- Setup scopes were not validated.
- Completeness was not validated against an independent inventory.
- Shards and fragments were not validated.
- Selector semantics were not changed or validated.
- The result does not prove 45k-test scale.
- It does prove schema-v1 codec/model handling and exact converted-subset preservation on a real
multi-thousand-test, high-edge-density dataset.
2 changes: 1 addition & 1 deletion smart-test-picker-cli/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ plugins {
}

group = 'com.sap.oss.smart-test-picker'
version = '0.1.0'
version = '0.2.0'

dependencies {
implementation project(':smart-test-picker-common')
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
name = "smart-test-picker",
description = "Regression test selection via per-test runtime coverage",
mixinStandardHelpOptions = true,
version = "0.1.0",
version = "0.2.0",
subcommands = {
ExecToXmlCommand.class,
GenerateMapCommand.class,
Expand Down
2 changes: 1 addition & 1 deletion smart-test-picker-common/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ plugins {
}

group = 'com.sap.oss.smart-test-picker'
version = '0.1.0'
version = '0.2.0'

dependencies {
implementation 'com.google.code.gson:gson:2.10.1'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
// SPDX-FileCopyrightText: 2024-2026 SAP SE or an SAP affiliate company and Smart Test Picker contributors
// SPDX-License-Identifier: Apache-2.0
package com.sap.oss.smarttestpicker.coverage;

/** Authoritative wire-contract versions owned by STP Core. */
public final class CoverageMapContract
{
public static final int SCHEMA_VERSION = 1;

private CoverageMapContract() {}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
// SPDX-FileCopyrightText: 2024-2026 SAP SE or an SAP affiliate company and Smart Test Picker contributors
// SPDX-License-Identifier: Apache-2.0
package com.sap.oss.smarttestpicker.coverage;

/** Authoritative execution-plan wire version owned by STP Core. */
public final class ExecutionPlanContract
{
public static final int VERSION = 1;

private ExecutionPlanContract() {}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
// SPDX-FileCopyrightText: 2024-2026 SAP SE or an SAP affiliate company and Smart Test Picker contributors
// SPDX-License-Identifier: Apache-2.0
package com.sap.oss.smarttestpicker.coverage.model;

import java.util.Set;
import java.util.TreeSet;

/** Independent expected test inventory and shard identities for one collection. */
public record CollectionExpectation(TestInventory testInventory, Set<ShardId> expectedShards)
{
public CollectionExpectation
{
if (testInventory == null) throw new IllegalArgumentException("test inventory is required");
expectedShards = Set.copyOf(new TreeSet<>(expectedShards));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
// SPDX-FileCopyrightText: 2024-2026 SAP SE or an SAP affiliate company and Smart Test Picker contributors
// SPDX-License-Identifier: Apache-2.0
package com.sap.oss.smarttestpicker.coverage.model;

public enum CollectionStatus { COLLECTED_WITH_COVERAGE, COLLECTED_EMPTY, COLLECTION_FAILED }
Loading
Loading