Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions TSF/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,16 +75,16 @@ For these files, and in particular the workflow files, caution must be exercised
Moreover, some parts of the documentation must be adapted to the new version.


### What can not be updated without further precautions?
### What cannot be updated without further precautions?

* ``cmake/ci.cmake``
This file defines, in particular, the various custom cmake targets; in particular, the various configurations for the execution of the unit- and integration-tests are defined.
The TSF requires, or, at the very least, strongly encourages us to collect test-results.
In order to do this efficiently, the ctest command is adapted to automatically generate the junit-logs of each test-run.
For this, the option ``--output-junit`` is set with output path ``../my_logs/TARGETNAME_junit.xml``, where TARGETNAME is replaced by the name of the respective cmake target; in case that this convention is insufficient to uniquely identify the logs, TARGETNAME is amended by a number.
When updating, it must be ensured that these adaptations are preserved.
Moreover, if the update introduces new cmake targets or new executions of ctest, it must be ensured, that the junit-log is generated and stored with a similar naming convention in the folder "../my_logs/".
Otherwise, it can not be ensured that the test data are accurately captured.
Moreover, if the update introduces new cmake targets or new executions of ctest, it must be ensured that the junit-log is generated and stored with a similar naming convention in the folder "../my_logs/".
Otherwise, it cannot be ensured that the test data are accurately captured.

* ``cmake/download_test_data.cmake``
This file is modified to ensure that the test-data are not downloaded from the original test-data repository, but instead from the copy of that repository within the Eclipse S-CORE organisation.
Expand All @@ -94,16 +94,16 @@ Moreover, some parts of the documentation must be adapted to the new version.
This file collects, in particular, the files containing the unit- and integration-tests in a list, which is given to cmake.
Custom tests were added in TSF/tests to document the fulfillment of the expectations.
To ensure that these tests are run, the file tests/CMakeLists.txt has been modified.
During the update, it must be ensured, that the custom tests are still being executed.
During the update, it must be ensured that the custom tests are still being executed.

* ``.github/workflows/parent-workflow.yml``
To ensure a specific execution order for the individual github workflows, their execution is orchestrated by the parent-workflow.
To ensure a specific execution order for the individual GitHub workflows, their execution is orchestrated by the parent-workflow.
To guarantee that this order is respected, it must be ensured that every workflow except for ``comment_check_amalgamation.yml``, ``docs-cleanup.yml``, ``parent-workflow.yml``, ``scorecards.yml`` and ``stale.yml`` runs ``on workflow_call``, only.

* ``.github/workflows/ubuntu.yml``
The ubuntu workflow orchestrates the parallel execution of various cmake targets with varying configurations running on the latest version of ubuntu.
The first TSF related adaptation of this workflow is that every step, in which a junit-report is generated, generates an artifact.
It must be ensured, that these artifacts are still generated after the update.
It must be ensured that these artifacts are still generated after the update.
The second adaptation is that the test-results are captured, processed and persistently stored or stored in the ubuntu-artifact.
Therefore, it must be ensured that the jobs ``publish_test_data_success``, ``publish_test_data_failure``, ``publish_test_data_cancellation`` and ``ubuntu_artifact`` are executed.
Moreover, in case that any further job is added by nlohmann, it must be ensured that this job is added to the list of jobs required before the latter workflows are executed.
Expand All @@ -126,7 +126,7 @@ Moreover, some parts of the documentation must be adapted to the new version.
For every workflow, it must be ensured that the conditions of their execution are unchanged.
The workflows ``check_amalgamation``, ``codeql``, ``dependency_review``, ``labeler`` and ``test_trudag_extensions`` generate an artifact, which must not be changed.
New workflows should be carefully reviewed.
If it is determined that their execution within the project is beneficial, and that they do not interfere with, then they should be integrated within the parent workflow at an appropriate place and their execution condition should be set to on ``workflow``, or their execution should be scheduled appropriately.
If it is determined that their execution within the project is beneficial, and that they do not interfere, then they should be integrated within the parent workflow at an appropriate place and their execution condition should be set to on ``workflow``, or their execution should be scheduled appropriately.
It is strongly recommended that the new workflow produces an artifact on success, and that the validator ``check_artifact_exists`` is adapted accordingly.
If nlohmann deletes any of the currently executed workflows, in particular ``check_amalgamation.yml``, ``codeql.yml``, ``dependency_review.yml``, ``labeler.yml``, ``test_trudag_extensions.yml`` and ``ubuntu.yml``, then it is strongly recommended to keep the currently executed version, since the automatic validator ``check_artifact_exists`` depends on the existence of these workflows.
In case that it is determined that these workflows should be deleted also in the documented copy of ``nlohmann/json``, then the validator ``check_artifact_exists`` and all its occurrences must be adapted accordingly.
Expand All @@ -144,7 +144,7 @@ The following adaptation is recommended, and has, unfortunately, not been automa


The following adaptations to the documentation have been automated; the python-script TSF/scripts/update_helper.py may be used to assist with these changes.
For the error-free execution is it necessary, however, to adhere to the naming scheme json_version_X_XX_X, and to not change the structure of the directories.
For error-free execution, it is, however, necessary to adhere to the naming scheme json_version_X_XX_X, and not to change the structure of the directories.

* ``TSF/Trustable/statements/JLS-11.md``
It must be ensured that the correct release date is used.
Expand All @@ -168,7 +168,7 @@ Based on the above observations, the following steps are recommended for each up
1. Merge branch master from the original nlohmann/json into an external fork of the eclipse-score/inc_nlohmann_json repository, where steps 2-12 shall be performed.
2. Confirm the deletion of cifuzz.yml, macos.yml and windows.yml.
3. Resolve the potential merge conflict in publish-documentation.yml by rejecting the incoming changes.
4. Update the versions of the github actions, if necessary.
4. Update the versions of the GitHub Actions, if necessary.
5. Resolve the potential merge conflicts in check_amalgamation.yml, codeql.yml, dependency_review.yml, labeler.yml and test_trudag_extensions.yml to ensure that the artifacts are generated, i.e. the jobs ``Generate XXX artifact`` and ``Upload XXX artifact`` are retained.
6. Resolve the potential merge conflict in ubuntu.yml following the above instructions.
7. Resolve the potential merge conflicts in cmake/download_test_data.cmake and cmake/ci.cmake following the above instructions.
Expand Down Expand Up @@ -197,7 +197,7 @@ The following should be considered:

* How has the algorithm for the accumulation of the trustable score changed? Ideally, it is not changed, otherwise the necessity for a new review arises.
* How has the data store interface changed? Ideally, it is not changed, but if it does and the expected schema format changes, data_store.py needs to be updated accordingly.
* How has the the expected configuration for the TSF items changed? It is known that this configuration changed (at least) once before. What does the potential change mean?
* How has the expected configuration for the TSF items changed? It is known that this configuration changed (at least) once before. What does the potential change mean?
* Do all custom references and validators as well as the data store interface work as before?
* Has the algorithm for the hashing changed, or are there any changes to the trustable scores? If so, investigate carefully!

Expand Down
6 changes: 3 additions & 3 deletions TSF/docs/ci_failure_rate_analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ This job was a short-lived Clang coverage experiment. In the selected time windo

### `ci_static_analysis_clang (ci_clang_tidy)`

In the analyzed period, most failed `ci_static_analysis_clang (ci_clang_tidy)` runs come from a few PRs that deliberately changed static analysis or CI tooling, plus real bug fix. In particular, PR [#4654](https://github.com/nlohmann/json/pull/4654) "Fix ~basic_json causing std::terminate", PR [#4663](https://github.com/nlohmann/json/pull/4663) "Add clang-tidy plugin to convert implicit conversions to explicit ones", and the change referenced as [#4701](https://github.com/nlohmann/json/pull/4701) "Suppress clang-analyzer-webkit.NoUncountedMemberChecker" all show multiple failed Ubuntu workflow runs while their clang-tidy / analyzer configuration was being tuned. Because this job treats every clang-tidy or analyzer diagnostic as a hard error, each new or stricter check initially makes the job fail until the warnings are fixed or suppressed. The resulting ~14 % failure rate is therefore explained by intentional tightening and maintenance of the static-analysis pipeline (plus normal PR iteration), rather than by unexplained or unaddressed misbehaviour in the library itself.
In the analysed period, most failed `ci_static_analysis_clang (ci_clang_tidy)` runs come from a few PRs that deliberately changed static analysis or CI tooling, plus real bug fixes. In particular, PR [#4654](https://github.com/nlohmann/json/pull/4654) "Fix ~basic_json causing std::terminate", PR [#4663](https://github.com/nlohmann/json/pull/4663) "Add clang-tidy plugin to convert implicit conversions to explicit ones", and the change referenced as [#4701](https://github.com/nlohmann/json/pull/4701) "Suppress clang-analyzer-webkit.NoUncountedMemberChecker" all show multiple failed Ubuntu workflow runs while their clang-tidy / analyzer configuration was being tuned. Because this job treats every clang-tidy or analyzer diagnostic as a hard error, each new or stricter check initially makes the job fail until the warnings are fixed or suppressed. The resulting ~14 % failure rate is therefore explained by intentional tightening and maintenance of the static-analysis pipeline (plus normal PR iteration), rather than by unexplained or unaddressed misbehaviour in the library itself.

### `ci_test_coverage`

Expand Down Expand Up @@ -103,7 +103,7 @@ branch / permission issues) rather than to the execution of the tests
themselves.

The job `clone_missing_labels` belongs to the separate “Labeler Workflow” and
synchronizes GitHub issue/PR labels for this repository with the organisation
synchronises GitHub issue/PR labels for this repository with the organisation
defaults. Its 2.63 % failure rate corresponds to 1 failed run out of 38,
and is related to repository/label management rather than to building or
testing the JSON library.
Expand All @@ -113,7 +113,7 @@ observed high failure rates were confined to publishing or repository-management

## Conclusion

Taken together, a period of there months show that all test-related
Taken together, a period of three months shows that all test-related
jobs in the Parent Workflow have 0 % failures, while the few non-zero
failure rates are confined to meta-jobs that handle publishing of historical
test data and label synchronization. This indicates a stable CI setup for
Expand Down
10 changes: 5 additions & 5 deletions TSF/docs/introduction/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,27 +28,27 @@ The JSON library by Niels Lohmann was developed with several design goals in min

- **Trivial Integration**: Comprising a single header file `json.hpp`, this library demands no complex build systems or dependencies, facilitating easy integration into any project using vanilla C++11.

- **Serious Testing**: Extensive unit tests ensure 100% coverage of the code, including exceptional behavior. Tools like Valgrind and Clang Sanitizers verify memory safety, while Google OSS-Fuzz performs continuous fuzz testing.
- **Serious Testing**: Extensive unit tests ensure 100% coverage of the code, including exceptional behaviour. Tools like Valgrind and Clang Sanitizers verify memory safety, while Google OSS-Fuzz performs continuous fuzz testing.

Notably, memory efficiency and speed were not primary focuses, allowing for slight trade-offs in these areas to prioritize ease of integration and reliability.

Baselibs Project Context
------------------------

The integration effort is situated within the baselibs project, aimed at qualifying library performance and compliance with internal standards. As part of this project, the TSF process has been embedded into the score repository to generate and analyze evidence regarding software trustability.
The integration effort is situated within the baselibs project, aimed at qualifying library performance and compliance with internal standards. As part of this project, the TSF process has been embedded into the score repository to generate and analyse evidence regarding software trustability.

Component Classification Strategy
-----------------------------------

- **Process Overview**: The baselibs project is upholding the TSF to define guidelines and generate reliable evidence of compliance, analyzing specific requirements such as MISRA and functionality consistency.
- **Process Overview**: The baselibs project is upholding the TSF to define guidelines and generate reliable evidence of compliance, analysing specific requirements such as MISRA and functionality consistency.

- **Challenges and Decisions**:
- Divergence from the original library code has been discussed to minimize unnecessary interference while adhering to project needs.
- MISRA compliance is selectively applied, and necessary adaptations are considered at the upstream level where applicable.

- **Strategic Directions**:
- Evidence requirements are mapped and analyzed for gaps, leading to possible code amendments or forks.
- Questions concerning the library's behavior are systematically answered, providing coverage details and tracing requirements to standards like ISO.
- Evidence requirements are mapped and analysed for gaps, leading to possible code amendments or forks.
- Questions concerning the library's behaviour are systematically answered, providing coverage details and tracing requirements to standards like ISO.

Find more descriptions on the ongoing process and requirements at `Eclipse Process Description <https://eclipse-score.github.io/process_description/main/trustable/index.html>`_.

Expand Down
2 changes: 1 addition & 1 deletion TSF/docs/nlohmann_closed_misbehaviours.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ issue-id | applies to S-CORE | problem | resolution | closed after
---------|-------------------|---------|------------|-------------
[2147](https://github.com/nlohmann/json/issues/2147) | No | Compilation error report for very old GCC (4.8.5) with `std::bind`/`std::function`. | Was labeled as a compiler bug; closed as stale after no further comment from author after trial of workaround. | 2.5 months
[2574](https://github.com/nlohmann/json/issues/2574) | No | `get<std::array<T,N>>()` fails for non-default-constructible element types. | Was fixed in https://github.com/nlohmann/json/pull/2576 and ticket closed after release in 3.10. | 3.5 months
[2655](https://github.com/nlohmann/json/issues/2655) | No | `std::pair` serialization expectation mismatch, object serialized instead of arrays. | Documentation updated to better explain behavior. | 5 months
[2655](https://github.com/nlohmann/json/issues/2655) | No | `std::pair` serialization expectation mismatch, object serialized instead of arrays. | Documentation updated to better explain behaviour. | 5 months
[2676](https://github.com/nlohmann/json/issues/2676) | No | NVCC warnings when instantiating templates under CUDA. | Was fixed in https://github.com/nlohmann/json/pull/3227 after release in 3.10.5. | 10 months
[2725](https://github.com/nlohmann/json/issues/2725) | No | Build with `-fno-exceptions` (and/or `JSON_NOEXCEPTIONS`) still hits `throw`. | Was already fixed in https://github.com/nlohmann/json/pull/2347. | the same day
[3025](https://github.com/nlohmann/json/issues/3025) | No | Brace-initialization in function calls was reported to select a copy path (`func({j});` prints `10` instead of `[10]`). | Was a duplicate of https://github.com/nlohmann/json/issues/2311 and that issue was declared not a bug and solved by referencing to Docu. | next day
Expand Down
2 changes: 1 addition & 1 deletion TSF/docs/non_reproducible_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Verifies that the library can be embedded directly into a CMake project using `a
- Builds multiple executables (`with_namespace_target`, `without_namespace_target`, `without_exceptions`)
- Each build creates object files, executables with embedded timestamps
- Subsequent runs create new artifacts with different timestamps
- CMake's incremental build cache persists between runs, causing different behavior
- CMake's incremental build cache persists between runs, causing different behaviour

---

Expand Down
4 changes: 2 additions & 2 deletions TSF/scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The following scripts are used in the automatic compilation of the trustable rep

## capture_test_data.py

The python-script [capture_test_data.py](capture_test_data.py) is intended to run at the end of the [ubuntu-workflow](../../.github/workflows/ubuntu.yml). It collects and combines the test-results that were collected in the individual jobs, appends the persistent test data storage and generates a database containing the results of the most recent tests only. Since storage space on the github is limited, the storage of test data is limited to two weeks only. It must be noted that this implementation is only intended as a temporary proof of concept!
The python-script [capture_test_data.py](capture_test_data.py) is intended to run at the end of the [ubuntu-workflow](../../.github/workflows/ubuntu.yml). It collects and combines the test-results that were collected in the individual jobs, appends the persistent test data storage and generates a database containing the results of the most recent tests only. Since storage space on GitHub is limited, the storage of test data is limited to two weeks only. It must be noted that this implementation is only intended as a temporary proof of concept!

## capture_test_data_memory_sensitive.py

Expand All @@ -23,7 +23,7 @@ Moreover, a size-check is performed on the persistent storage, using the followi
* It is expected, that less than 1,000 workflow runs, where a record of the test-result is triggered, happen per year, since these are only triggered once daily and on each push to main.
* It is expected, that a ten-year-record of test-results is sufficient for documentation purposes.

In view of these assumptions, we limit the storage to approximately 100,000 test-results and 100,000 workflow-metadata, which guarantees *en passant* that github's file size limit of 100MB is respected.
In view of these assumptions, we limit the storage to approximately 100,000 test-results and 100,000 workflow-metadata, which guarantees *en passant* that GitHub's file size limit of 100MB is respected.
In the worst case that every recorded test result is detected as a relevant change, this restraint collects test-results from 27 workflows in total.
Whenever either limit of 100,000 test-results and 100,000 workflow-metadata in the persistent data storage is violated, the script returns the error "The persistent data storage is too large! Please move persistent data to external storage.", thereby failing the workflow, which advises the maintainer to take up action.

Expand Down
Loading
Loading