Verification report#
score_coverage verification report
|
status: draft
security: NO
safety: ASIL_B
|
||||
This report is regenerated with every release. It doubles as the qualification verification report of the tool: the Tool Verification Report in the S-CORE platform documentation refers to it as the evidence of the validation.
Scope and environment#
Validated environment: Linux x86_64, Bazel 8.6.0, toolchains_llvm 1.8.0
with LLVM 22.1.7, score_toolchains_rust 0.10.0 (Ferrocene built by
ferrocene_toolchain_builder 1.3.1), Python 3.12 (rules_python 1.8.5), rules_rust 0.68.2-score.
gcov backend: score_bazel_cpp_toolchains 1.0.3 with GCC 12.2.0 on Linux,
gcovr 8.6. The QNX transport (QCC of QNX SDP 8.0, score_qnx_unit_tests
0.2.0 under QEMU) is the collection path of the communication repository
and is not exercised by this repository’s CI; it is validated on a consumer
(see the release notes of the validating release).
Test inventory#
Test target |
Cases |
Verifies |
|---|---|---|
|
20 |
merge_profraw, merge_no_data, merge_tool_error |
|
71 |
report_merged_profile, report_allowlist, report_baseline_zero, report_rlib_expansion, report_missing_baseline, report_relative_paths, report_outputs, report_unmapped, scope_transitive, instrumentation_hint |
|
21 |
gcov_merge, gcov_baseline, gcov_html, report_relative_paths, report_baseline_zero, report_allowlist, report_unmapped, report_outputs |
|
55 |
just_yaml, just_markers, just_unknown_id, just_platform, just_missing_file |
|
53 |
eff_metric, eff_stale, eff_branch_only, eff_path_match, eff_html, eff_gcovr |
|
63 |
gate_threshold, gate_metric, gate_unrounded, gate_exit_codes, gate_no_verdict, summary_first, artifacts |
|
19 |
summary_first |
|
17 |
scope_transitive, scope_excludes, scope_baseline_objects, scope_gcno, scope_platform |
|
1 |
gate_no_verdict |
|
27 |
validation_ground_truth, instrumentation_hint, scope_platform, report_baseline_zero, report_relative_paths, report_allowlist, report_unmapped, gcov_merge, gcov_baseline, gcov_html, gate_exit_codes, gate_no_verdict, just_unknown_id, artifacts, summary_first |
Requirement coverage#
The links from test cases to requirements are generated: every unit test class
carries @verifies(<tool_req ids>), which writes PartiallyVerifies,
TestType and DerivationTechnique into the JUnit XML of the test run, and
docs-as-code turns the results into testcase needs with back-links on the
requirements (testlink column below, with the execution result of each
case). End-to-end pytest functions attach the same metadata using the
score_pytest decorator. The links reflect the test run that preceded the
documentation build.
Four requirements are verified outside the pytest suites and therefore carry no generated link:
Transitive in-workspace sou... (tool_req__coverage_scope_transitive), External and generated sour... (tool_req__coverage_scope_excludes) and Baseline objects accompany ... (tool_req__coverage_scope_baseline_objects) are verified by the seventeen Starlark analysis tests in
tests/unit/starlark(rules_testing produces no test properties).Ground-truth validation (tool_req__coverage_validation_ground_truth) is verified by the end-to-end run
tests/end_to_end/run_end_to_end_test.sh(golden LCOV comparison, see below).
Structural coverage of the tool#
Measured with coverage.py through bazel coverage --combined_report=lcov and
gated in CI by //tools:self_coverage_gate (current ratchet 95 % lines,
87 % branches; target 100 % with documented deviations).
File |
Lines (C0) |
Branches (C1) |
|---|---|---|
|
95.65 % (198/207) |
88.78 % (87/98) |
|
95.09 % (523/550) |
81.16 % (224/276) |
|
98.56 % (205/208) |
92.59 % (75/81) |
|
98.35 % (238/242) |
94.56 % (139/147) |
|
98.35 % (119/121) |
93.65 % (59/63) |
|
91.95 % (354/385) |
84.97 % (164/193) |
Total |
95.56 % (1637/1713) |
87.18 % (748/858) |
Static analysis#
ruff (rule set of the S-CORE Python guideline: E, W, F, I, B, C90, UP, SIM, RET; McCabe ceiling 15), pylint and ty run as Bazel aspects with findings failing the build. Current state: zero findings. buildifier checks the Starlark, yamlfmt the workflows; copyright headers are checked on every file.
End-to-end validation#
tests/end_to_end/run_end_to_end_test.sh builds a consumer workspace with a
tested and an untested C++ library, a header-only library reached through
strip_include_prefix, a tested Rust library and an untested Rust binary, one
justified line, and asserts:
the gate fails at 100 % and passes at 10 % (effective and raw mode);
the HTML, the summary and the archive tree are produced, the summary also when the gate fails;
the untested C++ file and the untested Rust binary appear with
LH:0;the LCOV matches
expected_lcov.dat, a hand-derived ground truth, record by record;the justified line raises effective above raw coverage;
fault injection: a corrupt report and a non-numeric threshold exit 2, and a misspelt justification id is reported and does not raise the effective coverage.
Deviations#
Structural coverage of the Python is below 100 %. The remaining lines are error-handling and llvm-cov fallback paths in
reporter.pyandeffective_coverage.py; they are covered by the fault-injection checks of the integration test where they are reachable and will be closed or justified before the first qualified release.Starlark (
coverage_scope.bzl,reporter_wrapper.bzl) has no structural coverage tooling. The rule and aspect are verified by eight analysis tests and by the end-to-end run.The gcovr backend of
effective_coverage.pyis unit-tested against real gcovr 8.6 markup but is not reachable throughgenerate_coverage_htmlin this release (QNX flow, tooling issue #427).