Tool requirements#
Every requirement satisfies a use case or a process requirement and carries the potential errors it mitigates as tags. The verification report lists the tests that verify each requirement.
ID |
Title |
Tags |
Implemented |
|---|---|---|---|
Artifacts tree |
archive; ERR-09 |
YES |
|
Backend selection |
report; gcov; ERR-08 |
YES |
|
Branch-only justifications |
effective; ERR-05 |
YES |
|
gcovr HTML reports are supported |
effective |
YES |
|
Justified lines are visible in the HTML |
effective |
YES |
|
Effective coverage metric |
effective; ERR-05 |
YES |
|
Justifications match files at path-component boundaries |
effective; ERR-04; ERR-06 |
YES |
|
Stale justifications are reported and not counted |
effective; ERR-04 |
YES |
|
Exit codes |
gate; ERR-03 |
YES |
|
Gated metric |
gate; ERR-01 |
YES |
|
Broken input yields no verdict |
gate; ERR-03; ERR-08 |
YES |
|
Threshold from the environment |
gate; ERR-03 |
YES |
|
Unrounded comparison |
gate; ERR-03 |
YES |
|
gcov backend baseline from gcno notes |
report; gcov; ERR-01 |
YES |
|
gcov backend HTML and summary through gcovr |
report; gcov |
YES |
|
gcov backend merges per-test LCOV records by summation |
report; gcov; ERR-02 |
YES |
|
Instrumentation filter hint |
report; ERR-13 |
YES |
|
In-code markers |
justification; ERR-04 |
YES |
|
Justified locations exist |
justification; ERR-04 |
YES |
|
Platform filter |
justification |
YES |
|
Unknown marker ids do not justify anything |
justification; ERR-04 |
YES |
|
Justification YAML is validated |
justification; ERR-04 |
YES |
|
A test without instrumentation produces no coverage output |
collection |
YES |
|
Per-test profiles are merged with llvm-profdata |
collection; ERR-02 |
YES |
|
A missing or failing llvm-profdata fails the test's collection |
collection; ERR-02 |
YES |
|
The report is restricted to the scope allowlist |
report |
YES |
|
Untested in-scope files appear at exact 0 % |
report; ERR-01 |
YES |
|
One merged profile for all tests |
report; ERR-02 |
YES |
|
A missing baseline object is an error |
report; ERR-01 |
YES |
|
Report contents |
report |
YES |
|
Report paths are workspace-relative |
report; ERR-06 |
YES |
|
Baseline archives are reduced to members with a coverage mapping |
report; ERR-07 |
YES |
|
In-scope files without any coverage data are listed |
report; ERR-01 |
YES |
|
Baseline objects accompany the scope |
scope; ERR-01 |
YES |
|
External and generated sources are excluded from the scope |
scope |
YES |
|
gcno notes files accompany the scope |
scope; gcov; ERR-01 |
YES |
|
Transitive in-workspace sources define the scope |
scope; ERR-01 |
YES |
|
Summary is written before the verdict |
summary |
YES |
|
Ground-truth validation |
validation; ERR-02; ERR-07 |
YES |
Scope#
Transitive in-workspace sources define the scope
|
safety: ASIL_B
|
||||
|
|||||
External and generated sources are excluded from the scope
|
safety: QM
|
||||
|
|||||
gcno notes files accompany the scope
|
safety: ASIL_B
|
||||
|
|||||
Baseline objects accompany the scope
|
safety: ASIL_B
|
||||
|
|||||
Collection#
Per-test profiles are merged with llvm-profdata
|
safety: ASIL_B
|
||||
For each test, the merger shall merge all |
|||||
A test without instrumentation produces no coverage output
|
safety: QM
|
||||
When a test has no instrumented objects or no |
|||||
A missing or failing llvm-profdata fails the test's collection
|
safety: ASIL_B
|
||||
When |
|||||
Report#
One merged profile for all tests
|
safety: ASIL_B
|
||||
The reporter shall extract the |
|||||
The report is restricted to the scope allowlist
|
safety: ASIL_B
|
||||
The reporter shall exclude every file with coverage data that is not in the
allowlist from all three report formats, matching each excluded compiled
file exactly (an excluded |
|||||
Untested in-scope files appear at exact 0 %
|
safety: ASIL_B
|
||||
For every allowlisted file that appears in the baseline objects but in no
test binary, the reporter shall run |
|||||
In-scope files without any coverage data are listed
|
safety: ASIL_B
|
||||
An allowlisted file for which neither a test binary nor a baseline object
carries a coverage mapping (no translation unit includes it, or it holds
only template code that is never instantiated) cannot be rendered by
|
|||||
Baseline archives are reduced to members with a coverage mapping
|
safety: ASIL_B
|
||||
|
|||||
A missing baseline object is an error
|
safety: ASIL_B
|
||||
When an entry of the baseline objects manifest cannot be resolved to an existing file, the reporter shall exit non-zero instead of silently dropping the object. |
|||||
Report paths are workspace-relative
|
safety: QM
|
||||
The reporter shall name every file by its canonical path in all three
report formats: LCOV |
|||||
Report contents
|
safety: ASIL_B
|
||||
The reporter’s output shall be a zip containing |
|||||
Backend selection
|
safety: ASIL_B
|
||||
|
|||||
gcov backend merges per-test LCOV records by summation
|
safety: ASIL_B
|
||||
The gcov reporter shall read every per-test LCOV file Bazel’s collector
lists (ignoring |
|||||
gcov backend baseline from gcno notes
|
safety: ASIL_B
|
||||
For every allowlisted file that has no per-test data, the gcov reporter
shall run |
|||||
gcov backend HTML and summary through gcovr
|
safety: QM
|
||||
The gcov reporter shall render |
|||||
Justifications#
Justification YAML is validated
|
safety: ASIL_B
|
||||
The justification processor shall reject, with exit 1 and a message per
finding, a YAML that is not a mapping with an integer |
|||||
In-code markers
|
safety: ASIL_B
|
||||
The justification processor shall resolve |
|||||
Unknown marker ids do not justify anything
|
safety: ASIL_B
|
||||
A marker whose id is not in the (platform-filtered) YAML, a |
|||||
Platform filter
|
safety: QM
|
||||
When a platform is given, only justifications listing that platform shall apply. |
|||||
Justified locations exist
|
safety: ASIL_B
|
||||
A YAML location whose file does not exist shall be reported as an error and the processor shall exit 1 after writing the manifest. |
|||||
Effective coverage#
Effective coverage metric
|
safety: ASIL_B
|
||||
Effective line coverage shall be |
|||||
Stale justifications are reported and not counted
|
safety: ASIL_B
|
||||
A justified line that is covered in any instantiation and has no uncovered branch shall be reported as stale and shall not increase the effective coverage. |
|||||
Branch-only justifications
|
safety: ASIL_B
|
||||
A justified line that is covered but has a branch direction no instantiation covers shall count its truly uncovered directions once as justified branches and shall not count as a justified line. |
|||||
Justifications match files at path-component boundaries
|
safety: ASIL_B
|
||||
A justification for a file shall apply to a report page only when the page’s
source path equals the justified path or ends with it at a path-component
boundary; |
|||||
Justified lines are visible in the HTML
|
safety: QM
|
||||
Justified lines shall be restyled in the HTML report with a |
|||||
gcovr HTML reports are supported
|
safety: QM
|
||||
The post-processor shall detect gcovr |
|||||
Gate#
Threshold from the environment
|
safety: ASIL_B
|
||||
The threshold shall be read from |
|||||
Gated metric
|
safety: ASIL_B
|
||||
With |
|||||
Unrounded comparison
|
safety: ASIL_B
|
||||
The gate shall compare the unrounded percentage with the threshold; a value that prints as 100.00 but is below 100 shall fail a threshold of 100. |
|||||
Exit codes
|
safety: ASIL_B
|
||||
|
|||||
Broken input yields no verdict
|
safety: ASIL_B
|
||||
A missing or non-zip coverage report, a report without |
|||||
Summary and archive#
Summary is written before the verdict
|
safety: QM
|
||||
The markdown summary ( |
|||||
Artifacts tree
|
safety: ASIL_B
|
||||
With |
|||||
Instrumentation filter hint
|
safety: ASIL_B
|
||||
When an in-scope file has no test data, no file of its directory has test
data, and a |
|||||
Validation#
Ground-truth validation
|
safety: ASIL_B
|
||||
The pipeline shall be validated end to end against a fixture workspace with
C++ and Rust units whose line and branch counts are derived by hand: the
produced LCOV shall match the expected records exactly ( |
|||||