Verification report#
score_coverage verification report
|
status: draft
security: NO
safety: ASIL_B
|
||||
This report is regenerated with every release. It doubles as the qualification verification report of the tool: the Tool Verification Report in the S-CORE platform documentation refers to it as the evidence of the validation.
Scope and environment#
Validated environment: Linux x86_64, Bazel 8.6.0, toolchains_llvm 1.8.0
with LLVM 22.1.7, score_toolchains_rust 0.10.0 (Ferrocene built by
ferrocene_toolchain_builder 1.3.1), Python 3.12 (rules_python 1.8.5), rules_rust 0.68.2-score.
gcov backend: score_bazel_cpp_toolchains 1.0.3 with GCC 12.2.0 on Linux,
gcovr 8.6. The QNX transport (QCC of QNX SDP 8.0, score_qnx_unit_tests
0.2.0 under QEMU) is the collection path of the communication repository
and is not exercised by this repository’s CI; it is validated on a consumer
(see the release notes of the validating release).
Test inventory#
Test target |
Cases |
Verifies |
|---|---|---|
|
20 |
merge_profraw, merge_no_data, merge_tool_error |
|
71 |
report_merged_profile, report_allowlist, report_baseline_zero, report_rlib_expansion, report_missing_baseline, report_relative_paths, report_outputs, report_unmapped, scope_transitive, instrumentation_hint |
|
21 |
gcov_merge, gcov_baseline, gcov_html, report_relative_paths, report_baseline_zero, report_allowlist, report_unmapped, report_outputs |
|
55 |
just_yaml, just_markers, just_unknown_id, just_platform, just_missing_file |
|
53 |
eff_metric, eff_stale, eff_branch_only, eff_path_match, eff_html, eff_gcovr |
|
63 |
gate_threshold, gate_metric, gate_unrounded, gate_exit_codes, gate_no_verdict, summary_first, artifacts |
|
19 |
summary_first |
|
14 |
scope_transitive, scope_excludes, scope_baseline_objects, scope_gcno |
|
31 |
validation_ground_truth, instrumentation_hint, report_baseline_zero, report_relative_paths, report_allowlist, report_unmapped, gcov_merge, gcov_baseline, gcov_html, gate_metric, gate_exit_codes, gate_no_verdict, just_markers, just_unknown_id, artifacts, summary_first |
Requirement coverage#
The links from test cases to requirements are generated: each pytest function
uses the score_pytest metadata decorator to write PartiallyVerifies,
TestType and DerivationTechnique into the JUnit XML. Docs-as-code turns
the results into testcase needs with back-links on the requirements
(testlink column below, with the execution result of each case). The links
reflect the test run that preceded the documentation build.
Three requirements are verified outside the pytest suites and therefore carry no generated link:
Transitive in-workspace sou... (tool_req__coverage_scope_transitive), External and generated sour... (tool_req__coverage_scope_excludes) and Baseline objects accompany ... (tool_req__coverage_scope_baseline_objects) are verified by the fourteen Starlark analysis tests in
score_coverage/tests/starlark(rules_testing produces no test properties).
Structural coverage of the tool#
Measured with coverage.py through bazel coverage --combined_report=lcov and
gated in CI by //tools:self_coverage_gate (current ratchet 95 % lines,
87 % branches; target 100 % with documented deviations).
File |
Lines (C0) |
Branches (C1) |
|---|---|---|
|
95.65 % (198/207) |
88.78 % (87/98) |
|
95.09 % (523/550) |
81.16 % (224/276) |
|
98.56 % (205/208) |
92.59 % (75/81) |
|
98.35 % (238/242) |
94.56 % (139/147) |
|
98.35 % (119/121) |
93.65 % (59/63) |
|
91.95 % (354/385) |
84.97 % (164/193) |
Total |
95.56 % (1637/1713) |
87.18 % (748/858) |
Static analysis#
ruff (rule set of the S-CORE Python guideline: E, W, F, I, B, C90, UP, SIM, RET; McCabe ceiling 15), pylint and ty run as Bazel aspects with findings failing the build. Current state: zero findings. buildifier checks the Starlark, yamlfmt the workflows; copyright headers are checked on every file.
End-to-end validation#
//tools/integration_tests:blackbox_test runs named pytest cases against a
copied consumer workspace with a tested and an untested C++ library, a
header-only library reached through strip_include_prefix, a tested Rust
library and an untested Rust binary. Separate LLVM and gcov fixtures collect
their reports before cases exercise:
effective and raw gate thresholds, with parametrized pass and fail values;
Markdown summaries, archive contents, working HTML and stylesheet links, canonical source paths, and exclusion of forwarded external code;
unmapped-file categories, exact zero-count baselines, and hand-derived LLVM and gcov LCOV ground truths;
the gcov warning and zero-count fallback when a narrow instrumentation filter omits a tested library;
fault injection for corrupt reports, invalid thresholds and unknown justification markers.
The pytest JUnit XML records a test case and requirement metadata for every scenario, including each parametrized gate value.
Deviations#
Structural coverage of the Python is below 100 %. The remaining lines are error-handling and llvm-cov fallback paths in
reporter.pyandeffective_coverage.py; they are covered by the fault-injection checks of the black-box test where they are reachable and will be closed or justified before the first qualified release.Starlark (
coverage_scope.bzl,reporter_wrapper.bzl) has no structural coverage tooling. The rule and aspect are verified by fourteen analysis tests and by the end-to-end run.The gcovr backend of
effective_coverage.pyis unit-tested against real gcovr 8.6 markup but is not reachable throughgenerate_coverage_htmlin this release (QNX flow, tooling issue #427).