Verification report#
score_coverage verification report
|
status: draft
security: NO
safety: ASIL_B
|
||||
This report is regenerated with every release. It doubles as the qualification verification report of the tool: the Tool Verification Report in the S-CORE platform documentation refers to it as the evidence of the validation.
Scope and environment#
Validated environment: Linux x86_64, Bazel 8.6.0, toolchains_llvm 1.8.0
with LLVM 22.1.7, score_toolchains_rust 0.10.0 (Ferrocene built by
ferrocene_toolchain_builder 1.3.1), Python 3.12 (rules_python 1.8.5), rules_rust 0.68.2-score.
gcov backend: score_bazel_cpp_toolchains 1.0.3 with GCC 12.2.0 on Linux,
gcovr 8.6. The QNX transport (QCC of QNX SDP 8.0, score_qnx_unit_tests
0.2.0 under QEMU) is the collection path of the communication repository
and is not exercised by this repository’s CI; it is validated on a consumer
(see the release notes of the validating release).
Test inventory#
Test target |
Cases |
Verifies |
|---|---|---|
|
20 |
merge_profraw, merge_no_data, merge_tool_error |
|
71 |
report_merged_profile, report_allowlist, report_baseline_zero, report_rlib_expansion, report_missing_baseline, report_relative_paths, report_outputs, report_unmapped, scope_transitive, instrumentation_hint |
|
21 |
gcov_merge, gcov_baseline, gcov_html, report_relative_paths, report_baseline_zero, report_allowlist, report_unmapped, report_outputs |
|
55 |
just_yaml, just_markers, just_unknown_id, just_platform, just_missing_file |
|
53 |
eff_metric, eff_stale, eff_branch_only, eff_path_match, eff_html, eff_gcovr |
|
63 |
gate_threshold, gate_metric, gate_unrounded, gate_exit_codes, gate_no_verdict, summary_first, artifacts |
|
19 |
summary_first |
|
14 |
scope_transitive, scope_excludes, scope_baseline_objects, scope_gcno |
|
25 |
validation_ground_truth, instrumentation_hint, report_baseline_zero, report_relative_paths, report_allowlist, report_unmapped, gcov_merge, gcov_baseline, gcov_html, gate_exit_codes, gate_no_verdict, just_unknown_id, artifacts, summary_first |
Requirement coverage#
The links from test cases to requirements are generated: every unit test class
carries @verifies(<tool_req ids>), which writes PartiallyVerifies,
TestType and DerivationTechnique into the JUnit XML of the test run, and
docs-as-code turns the results into testcase needs with back-links on the
requirements (testlink column below, with the execution result of each
case). The links reflect the test run that preceded the documentation build.
Four requirements are verified outside the pytest suites and therefore carry no generated link:
Transitive in-workspace sou... (tool_req__coverage_scope_transitive), External and generated sour... (tool_req__coverage_scope_excludes) and Baseline objects accompany ... (tool_req__coverage_scope_baseline_objects) are verified by the eleven Starlark analysis tests in
score_coverage/tests/starlark(rules_testing produces no test properties).Ground-truth validation (tool_req__coverage_validation_ground_truth) is verified by the end-to-end run
integration_tests/run_integration_test.sh(golden LCOV comparison, see below).
Structural coverage of the tool#
Measured with coverage.py through bazel coverage --combined_report=lcov and
gated in CI by //tools:self_coverage_gate (current ratchet 95 % lines,
87 % branches; target 100 % with documented deviations).
File |
Lines (C0) |
Branches (C1) |
|---|---|---|
|
95.65 % (198/207) |
88.78 % (87/98) |
|
95.09 % (523/550) |
81.16 % (224/276) |
|
98.56 % (205/208) |
92.59 % (75/81) |
|
98.35 % (238/242) |
94.56 % (139/147) |
|
98.35 % (119/121) |
93.65 % (59/63) |
|
91.95 % (354/385) |
84.97 % (164/193) |
Total |
95.56 % (1637/1713) |
87.18 % (748/858) |
Static analysis#
ruff (rule set of the S-CORE Python guideline: E, W, F, I, B, C90, UP, SIM, RET; McCabe ceiling 15), pylint and ty run as Bazel aspects with findings failing the build. Current state: zero findings. buildifier checks the Starlark, yamlfmt the workflows; copyright headers are checked on every file.
End-to-end validation#
integration_tests/run_integration_test.sh builds a consumer workspace with a
tested and an untested C++ library, a header-only library reached through
strip_include_prefix, a tested Rust library and an untested Rust binary, one
justified line, and asserts:
the gate fails at 100 % and passes at 10 % (effective and raw mode);
the HTML, the summary and the archive tree are produced, the summary also when the gate fails;
the untested C++ file and the untested Rust binary appear with
LH:0;the LCOV matches
expected_lcov.dat, a hand-derived ground truth, record by record;the justified line raises effective above raw coverage;
fault injection: a corrupt report and a non-numeric threshold exit 2, and a misspelt justification id is reported and does not raise the effective coverage.
Deviations#
Structural coverage of the Python is below 100 %. The remaining lines are error-handling and llvm-cov fallback paths in
reporter.pyandeffective_coverage.py; they are covered by the fault-injection checks of the integration test where they are reachable and will be closed or justified before the first qualified release.Starlark (
coverage_scope.bzl,reporter_wrapper.bzl) has no structural coverage tooling. The rule and aspect are verified by eight analysis tests and by the end-to-end run.The gcovr backend of
effective_coverage.pyis unit-tested against real gcovr 8.6 markup but is not reachable throughgenerate_coverage_htmlin this release (QNX flow, tooling issue #427).