Verification report#

score_coverage verification report
status: draft
security: NO
safety: ASIL_B
version: 1

This report is regenerated with every release. It doubles as the qualification verification report of the tool: the Tool Verification Report in the S-CORE platform documentation refers to it as the evidence of the validation.

Scope and environment#

Validated environment: Linux x86_64, Bazel 8.6.0, toolchains_llvm 1.8.0 with LLVM 22.1.7, score_toolchains_rust 0.10.0 (Ferrocene built by ferrocene_toolchain_builder 1.3.1), Python 3.12 (rules_python 1.8.5), rules_rust 0.68.2-score. gcov backend: score_bazel_cpp_toolchains 1.0.3 with GCC 12.2.0 on Linux, gcovr 8.6. The QNX transport (QCC of QNX SDP 8.0, score_qnx_unit_tests 0.2.0 under QEMU) is the collection path of the communication repository and is not exercised by this repository’s CI; it is validated on a consumer (see the release notes of the validating release).

Test inventory#

Test target

Cases

Verifies

//tests/unit/score_coverage:merger_test

20

merge_profraw, merge_no_data, merge_tool_error

//tests/unit/score_coverage:reporter_test

74

report_merged_profile, report_allowlist, report_baseline_zero, report_rlib_expansion, report_missing_baseline, report_relative_paths, report_outputs, report_unmapped, scope_transitive, instrumentation_hint

//tests/unit/score_coverage:gcov_reporter_test

21

gcov_merge, gcov_baseline, gcov_html, report_relative_paths, report_baseline_zero, report_allowlist, report_unmapped, report_outputs

//tests/unit/score_coverage:justify_test

55

just_yaml, just_markers, just_unknown_id, just_platform, just_missing_file

//tests/unit/score_coverage:effective_coverage_test

53

eff_metric, eff_stale, eff_branch_only, eff_path_match, eff_html, eff_gcovr

//tests/unit/score_coverage:generate_coverage_html_test

63

gate_threshold, gate_metric, gate_unrounded, gate_exit_codes, gate_no_verdict, summary_first, artifacts

//tests/unit/score_coverage:coverage_summary_test

19

summary_first

//tests/unit/starlark:coverage_scope_tests (18 analysis tests)

18

scope_transitive, scope_excludes, scope_baseline_objects, scope_gcno, scope_platform

//tests/end_to_end/tests:blackbox_test

38

gate_no_verdict, gate_threshold, gate_exit_codes, summary_first, validation_ground_truth, report_baseline_zero, report_merged_profile, scope_platform, report_relative_paths, report_allowlist, report_unmapped, report_outputs, artifacts, eff_metric, just_unknown_id

//tests/end_to_end/tests:gcov_test

9

validation_ground_truth, instrumentation_hint, scope_platform, report_unmapped, gcov_baseline, gcov_html, eff_gcovr, gate_exit_codes, artifacts, summary_first

//tests/end_to_end/tests:cpp_semantics_test

88 (85 passed, three strict expected failures: two for LLVM exception unwinding and one for LLVM scope filtering, issue #70)

validation_ground_truth, report_allowlist, report_merged_profile, gcov_merge: shared inline/template code across translation units and template instances, exception paths, uncalled code and counter summation across tests, with both LLVM and gcov

The test support code is checked separately by //tests/end_to_end/test_framework:coverage_checks_test (25 cases). These unit tests validate CHECK parsing and rejection of invalid expectations and duplicate LCOV counters. They test the validation machinery itself and do not claim verification of product requirements.

Requirement coverage#

The links from test cases to requirements are generated: every unit test class carries @verifies(<tool_req ids>), which writes PartiallyVerifies, TestType and DerivationTechnique into the JUnit XML of the test run, and docs-as-code turns the results into testcase needs with back-links on the requirements (testlink column below, with the execution result of each case). End-to-end pytest functions attach the same metadata using the score_pytest decorator. The links reflect the test run that preceded the documentation build.

Requirements and the tests that verify them#

ID

Title

Testlink

tool_req__coverage_artifacts

Artifacts tree

tool_req__coverage_backend_select

Backend selection

tool_req__coverage_eff_branch_only

Branch-only justifications

tool_req__coverage_eff_gcovr

gcovr HTML reports are supported

tool_req__coverage_eff_html

Justified lines are visible in the HTML

tool_req__coverage_eff_metric

Effective coverage metric

tool_req__coverage_eff_path_match

Justifications match files at path-component boundaries

tool_req__coverage_eff_stale

Stale justifications are reported and not counted

tool_req__coverage_gate_exit_codes

Exit codes

tool_req__coverage_gate_metric

Gated metric

tool_req__coverage_gate_no_verdict

Broken input yields no verdict

tool_req__coverage_gate_threshold

Threshold from the environment

tool_req__coverage_gate_unrounded

Unrounded comparison

tool_req__coverage_gcov_baseline

gcov backend baseline from gcno notes

tool_req__coverage_gcov_html

gcov backend HTML and summary through gcovr

tool_req__coverage_gcov_merge

gcov backend merges per-test LCOV records by summation

tool_req__coverage_instrumentation_hint

Instrumentation filter hint

tool_req__coverage_just_markers

In-code markers

tool_req__coverage_just_missing_file

Justified locations exist

tool_req__coverage_just_platform

Platform filter

tool_req__coverage_just_unknown_id

Unknown marker ids do not justify anything

tool_req__coverage_just_yaml

Justification YAML is validated

tool_req__coverage_merge_no_data

A test without instrumentation produces no coverage output

tool_req__coverage_merge_profraw

Per-test profiles are merged with llvm-profdata

tool_req__coverage_merge_tool_error

A missing or failing llvm-profdata fails the test's collection

tool_req__coverage_report_allowlist

The report is restricted to the scope allowlist

tool_req__coverage_report_baseline_zero

Untested in-scope files appear at exact 0 %

tool_req__coverage_report_merged_profile

One merged profile for all tests

tool_req__coverage_report_missing_baseline

A missing baseline object is an error

tool_req__coverage_report_outputs

Report contents

tool_req__coverage_report_relative_paths

Report paths are workspace-relative

tool_req__coverage_report_rlib_expansion

Baseline archives are reduced to members with a coverage mapping

tool_req__coverage_report_unmapped

In-scope files without any coverage data are listed

tool_req__coverage_scope_baseline_objects

Baseline objects accompany the scope

tool_req__coverage_scope_excludes

External and generated sources are excluded from the scope

tool_req__coverage_scope_gcno

gcno notes files accompany the scope

tool_req__coverage_scope_platform

Scope evaluated for the run's platform

tool_req__coverage_scope_transitive

Transitive in-workspace sources define the scope

tool_req__coverage_summary_first

Summary is written before the verdict

tool_req__coverage_validation_ground_truth

Ground-truth validation

Three requirements are verified outside the pytest suites and therefore carry no generated link:

The LLVM and gcov ground-truth comparisons both link to Ground-truth validation (tool_req__coverage_validation_ground_truth) through pytest.

Test results of the linked test cases

Structural coverage of the tool#

Measured with coverage.py through bazel coverage --combined_report=lcov and gated in CI by //tools:self_coverage_gate (current ratchet 95 % lines, 87 % branches; target 100 % with documented deviations).

File

Lines (C0)

Branches (C1)

score_coverage/coverage_summary.py

95.65 % (198/207)

88.78 % (87/98)

score_coverage/effective_coverage.py

95.09 % (523/550)

81.16 % (224/276)

score_coverage/generate_coverage_html.py

98.56 % (205/208)

92.59 % (75/81)

score_coverage/justify.py

98.35 % (238/242)

94.56 % (139/147)

score_coverage/merger.py

98.35 % (119/121)

93.65 % (59/63)

score_coverage/reporter.py

91.95 % (354/385)

84.97 % (164/193)

Total

95.56 % (1637/1713)

87.18 % (748/858)

Static analysis#

ruff (rule set of the S-CORE Python guideline: E, W, F, I, B, C90, UP, SIM, RET; McCabe ceiling 15), pylint and ty run as Bazel aspects with findings failing the build. Current state: zero findings. buildifier checks the Starlark, yamlfmt the workflows; copyright headers are checked on every file.

End-to-end validation#

The pytest targets build a consumer workspace with tested and untested C++ and Rust sources, a header-only library reached through strip_include_prefix, and one justified line. Bazel schedules the gcov target exclusively because the backends write to the same consumer report path.

The end-to-end pytest cases verify that the LLVM gate fails at 100 % and passes at 10 %, with and without justifications. They also verify Markdown summaries, including summary output when the gate fails. CLI cases use temporary workspaces to check exit 2 and diagnostics for missing or corrupt reports and an invalid threshold. An unknown marker ID emits a warning and leaves effective coverage equal to raw coverage. Cases needing a real LLVM report reuse one coverage build per pytest session. The LLVM LCOV cases read that build’s report without additional CLI calls: they compare all line and branch records with expected_lcov.dat and check zero-hit baselines, measured C++ and Rust sources (including a library tested from a subpackage), host platform selection, canonical vendored-header paths, exclusion of forwarded third-party code, and absence of invented coverage for an uncompiled header.

The LLVM artifact run also verifies that the effective gate passes at 10 %. The HTML cases unpack the CI ZIP to check zero-coverage line totals against LCOV, portable source and stylesheet links, canonical paths, and scope filtering. The archive checks compare the complete list of unmapped-source categories and verify the increase from raw to effective coverage. The raw gate at 10 % shares the HTML output check without justifications, and the effective gate at 100 % shares the failing summary case.

The gcov cases verify the gate at 100 % and 10 %, compare archived LCOV with the hand-derived expected_lcov_gcov.dat, and check target platform selection, archive navigation, unmapped sources, and effective coverage. The cases share one coverage build and read its archived results. The instrumentation filter case runs coverage with a narrower filter and checks the warning and the cross-package library’s zero-hit baseline.

Deviations#

  • Structural coverage of the Python is below 100 %. The remaining lines are error-handling and llvm-cov fallback paths in reporter.py and effective_coverage.py; they are covered by the fault-injection checks of the integration test where they are reachable and will be closed or justified before the first qualified release.

  • Starlark (coverage_scope.bzl, reporter_wrapper.bzl) has no structural coverage tooling. The rule and aspect are verified by eight analysis tests and by the end-to-end run.