Verification report#
score_coverage verification report
|
status: draft
security: NO
safety: ASIL_B
|
||||
This report is regenerated with every release. It doubles as the qualification verification report of the tool: the Tool Verification Report in the S-CORE platform documentation refers to it as the evidence of the validation.
Scope and environment#
Validated environment: Linux x86_64, Bazel 8.6.0, toolchains_llvm 1.8.0
with LLVM 22.1.7, score_toolchains_rust 0.10.0 (Ferrocene built by
ferrocene_toolchain_builder 1.3.1), Python 3.12 (rules_python 1.8.5), rules_rust 0.68.2-score.
gcov backend: score_bazel_cpp_toolchains 1.0.3 with GCC 12.2.0 on Linux,
gcovr 8.6. The QNX transport (QCC of QNX SDP 8.0, score_qnx_unit_tests
0.2.0 under QEMU) is the collection path of the communication repository
and is not exercised by this repository’s CI; it is validated on a consumer
(see the release notes of the validating release).
Test inventory#
Test target |
Cases |
Verifies |
|---|---|---|
|
20 |
merge_profraw, merge_no_data, merge_tool_error |
|
74 |
report_merged_profile, report_allowlist, report_baseline_zero, report_rlib_expansion, report_missing_baseline, report_relative_paths, report_outputs, report_unmapped, scope_transitive, instrumentation_hint |
|
21 |
gcov_merge, gcov_baseline, gcov_html, report_relative_paths, report_baseline_zero, report_allowlist, report_unmapped, report_outputs |
|
55 |
just_yaml, just_markers, just_unknown_id, just_platform, just_missing_file |
|
53 |
eff_metric, eff_stale, eff_branch_only, eff_path_match, eff_html, eff_gcovr |
|
63 |
gate_threshold, gate_metric, gate_unrounded, gate_exit_codes, gate_no_verdict, summary_first, artifacts |
|
19 |
summary_first |
|
18 |
scope_transitive, scope_excludes, scope_baseline_objects, scope_gcno, scope_platform |
|
38 |
gate_no_verdict, gate_threshold, gate_exit_codes, summary_first, validation_ground_truth, report_baseline_zero, report_merged_profile, scope_platform, report_relative_paths, report_allowlist, report_unmapped, report_outputs, artifacts, eff_metric, just_unknown_id |
|
9 |
validation_ground_truth, instrumentation_hint, scope_platform, report_unmapped, gcov_baseline, gcov_html, eff_gcovr, gate_exit_codes, artifacts, summary_first |
|
88 (85 passed, three strict expected failures: two for LLVM exception unwinding and one for LLVM scope filtering, issue #70) |
validation_ground_truth, report_allowlist, report_merged_profile, gcov_merge: shared inline/template code across translation units and template instances, exception paths, uncalled code and counter summation across tests, with both LLVM and gcov |
The test support code is checked separately by
//tests/end_to_end/test_framework:coverage_checks_test (25 cases). These unit tests
validate CHECK parsing and rejection of invalid expectations and duplicate LCOV
counters. They test the validation machinery itself and do not claim verification
of product requirements.
Requirement coverage#
The links from test cases to requirements are generated: every unit test class
carries @verifies(<tool_req ids>), which writes PartiallyVerifies,
TestType and DerivationTechnique into the JUnit XML of the test run, and
docs-as-code turns the results into testcase needs with back-links on the
requirements (testlink column below, with the execution result of each
case). End-to-end pytest functions attach the same metadata using the
score_pytest decorator. The links reflect the test run that preceded the
documentation build.
Three requirements are verified outside the pytest suites and therefore carry no generated link:
Transitive in-workspace sou... (tool_req__coverage_scope_transitive), External and generated sour... (tool_req__coverage_scope_excludes) and Baseline objects accompany ... (tool_req__coverage_scope_baseline_objects) are verified by the seventeen Starlark analysis tests in
tests/unit/starlark(rules_testing produces no test properties).
The LLVM and gcov ground-truth comparisons both link to Ground-truth validation (tool_req__coverage_validation_ground_truth) through pytest.
Structural coverage of the tool#
Measured with coverage.py through bazel coverage --combined_report=lcov and
gated in CI by //tools:self_coverage_gate (current ratchet 95 % lines,
87 % branches; target 100 % with documented deviations).
File |
Lines (C0) |
Branches (C1) |
|---|---|---|
|
95.65 % (198/207) |
88.78 % (87/98) |
|
95.09 % (523/550) |
81.16 % (224/276) |
|
98.56 % (205/208) |
92.59 % (75/81) |
|
98.35 % (238/242) |
94.56 % (139/147) |
|
98.35 % (119/121) |
93.65 % (59/63) |
|
91.95 % (354/385) |
84.97 % (164/193) |
Total |
95.56 % (1637/1713) |
87.18 % (748/858) |
Static analysis#
ruff (rule set of the S-CORE Python guideline: E, W, F, I, B, C90, UP, SIM, RET; McCabe ceiling 15), pylint and ty run as Bazel aspects with findings failing the build. Current state: zero findings. buildifier checks the Starlark, yamlfmt the workflows; copyright headers are checked on every file.
End-to-end validation#
The pytest targets build a consumer workspace with tested and untested C++
and Rust sources, a header-only library reached through strip_include_prefix,
and one justified line. Bazel schedules the gcov target exclusively because
the backends write to the same consumer report path.
The end-to-end pytest cases verify that the LLVM gate fails at 100 % and
passes at 10 %, with and without justifications. They also verify Markdown
summaries, including summary output when the gate fails. CLI cases use
temporary workspaces to check exit 2 and diagnostics for missing or corrupt
reports and an invalid threshold. An unknown marker ID emits a warning and
leaves effective coverage equal to raw coverage. Cases needing a real LLVM
report reuse one coverage build per pytest session.
The LLVM LCOV cases read that build’s report without additional CLI calls:
they compare all line and branch records with expected_lcov.dat and
check zero-hit baselines, measured C++ and Rust sources (including a library
tested from a subpackage), host platform selection, canonical vendored-header
paths, exclusion of forwarded third-party code, and absence of invented
coverage for an uncompiled header.
The LLVM artifact run also verifies that the effective gate passes at 10 %. The HTML cases unpack the CI ZIP to check zero-coverage line totals against LCOV, portable source and stylesheet links, canonical paths, and scope filtering. The archive checks compare the complete list of unmapped-source categories and verify the increase from raw to effective coverage. The raw gate at 10 % shares the HTML output check without justifications, and the effective gate at 100 % shares the failing summary case.
The gcov cases verify the gate at 100 % and 10 %, compare archived LCOV with
the hand-derived expected_lcov_gcov.dat, and check target platform selection,
archive navigation, unmapped sources, and effective coverage. The cases share
one coverage build and read its archived results. The instrumentation filter
case runs coverage with a narrower filter and checks the warning and the
cross-package library’s zero-hit baseline.
Deviations#
Structural coverage of the Python is below 100 %. The remaining lines are error-handling and llvm-cov fallback paths in
reporter.pyandeffective_coverage.py; they are covered by the fault-injection checks of the integration test where they are reachable and will be closed or justified before the first qualified release.Starlark (
coverage_scope.bzl,reporter_wrapper.bzl) has no structural coverage tooling. The rule and aspect are verified by eight analysis tests and by the end-to-end run.