Test Case Coverage (test_case_coverage)

test_case_coverage verifies that a declared set of requirements is completely covered by automated test cases. Coverage is locked and confirmed by a committed YAML file;

Requirement IDs are extracted automatically and test records are read from the gtest.lobster file.

A test case is specified either by Given/When/Then or by a formal test specification in RST that it references with RecordProperty("test-spec", "<label>") (see Validation).

Note

For how to annotate tests with lobster-tracing see Validation.

The component rule integrates test case coverage when test_case_coverage_lock is provided:

component(
    name = "my_component",
    requirements  = [":my_component_requirements"],
    components    = [":unit_a", ":unit_b"],
    test_case_coverage_lock = "test_case_coverage.lock.yaml",
)

When the build reports a test_case_coverage drift failure, run (here on my_component):

bazel run //<pkg>:<name>.update

Two phases

test_case_coverage has two distinct operational phases:

Test phase — bazel build //... / bazel test //...

test_runner is invoked as a build action inside the dependable_element rule for each component that has test_case_coverage_lock set. The effective enforcement stringency is controlled by the enclosing dependable_element’s maturity attribute:

• maturity = "development" — --allow-check-failures is passed so the .lobster artifact is always produced even when coverage drifts. Both lock-drift failures and missing Given/Then errors are downgraded to warnings;

• maturity = "release" — no --allow-check-failures; a drift or a missing Given or Then fails the Bazel build action directly. The .lobster artifact is physically written before the action exits, but Bazel marks the action as failed.

Run phase — bazel run //<pkg>:<name>.update

Computes the lock from the current gtest.lobster and overwrites the committed test_case_coverage.lock.yaml in the workspace. Committing the result constitutes approval of test case coverage.

Architecture

bazel/rules/rules_score/src/test_case_coverage/
├── read_gtest_lobster.py       # reads gtest.lobster; also resolve_path, scan, req_ids
├── parse_test_spec.py          # test_specification rule: validates RST, writes JSON
├── test_spec.py                # loads test_specification JSON by label
├── compute_lock.py             # uid/spec serialisation + YAML lock
├── check_lock.py               # committed vs computed comparison
├── lobster_generator.py        # emits LOBSTER activity artifact
├── update_runner.py            # entry point: bazel run <component>.update
└── test_runner.py              # entry point: bazel test //...

Data flow

Requirements targets (.lobster) ──► extract requirement IDs
subrule_lobster_gtest ───────────► gtest.lobster
test_specification (RST) ─► parse_test_spec ─► <name>.test_spec.json
                                           │                 │
                                 read_gtest_lobster      test_spec
                                 (uid, req refs, GWT,    (spec by label,
                                  test-spec label)        via manifest)
                                           │                 │
                                     compute_lock ◄──────────┘
                                 ┌─────────┴──────────────┐
                           RUN phase                TEST phase
                       (bazel run .update)       (bazel test)
                                 │                    │
                          rewrite lock         compare uid+spec
                          file in WS           vs committed lock
                                                      │
                                            lobster_generator
                                            (ok / fail per test)

Lock file format

Auto-generated by the .update target. The engineer commits it after reviewing git diff.

schema_version: 3
requirements:
  - id: MessagePassing.OsIpcFaultHandling
    test_cases:
      - uid:   "//score/message_passing/ConnectionSuite:OsIpcFaultHandlingTest"
        given: a connected client
        when:  the OS IPC call fails
        then:  the client receives an error
      - uid:   "//score/message_passing/ConnectionSuite:OsIpcReconnectTest"
        spec:
          ref:   //score/message_passing/test_specs:reconnect
          goal:  Show that a client reconnects after the server restarts.
          focus: fault injection
          steps:
            - precondition: a connected client
              action:       restart the server
              expected:     the client reconnects within 1 s

Field	Notes
requirements[].id	Matches lobster-tracing values; extracted from requirements targets
test_cases[].uid	//bazel_package/SuiteName:TestName — package-scoped gtest tag
test_cases[].given / when / then	Individual GWT fields from RecordProperty annotations or the test name; any change makes the lock stale. given and then must be non-blank. when may be empty and is then stored as ''
test_cases[].spec	Formal test specification of a test case that sets test-spec; replaces given/when/then (an entry with both is rejected). Holds ref (the test-spec reference as written in the test), goal, focus and steps (precondition, action, expected) as parsed from the RST; any change makes the lock stale. Multi-line text is written as a YAML literal block

Upgrading lobster to a version that derives the GWT fields from test names changes the text of tests that follow the naming pattern and set no properties. Run .update and review the diff, see Validation.

Test specifications

test_specification targets run parse_test_spec as the build action TestSpecificationCheck: a malformed RST fails the build regardless of maturity. The action writes <name>.test_spec.json with the parsed content and the labels of the target and of its RST file, relative to their own repository (//pkg:name, //pkg:name.rst).

The test case references a spec as text in a RecordProperty, so Bazel’s repository mapping does not apply to it. Instead, the unit macro records how each test_specs entry is spelled in the unit’s repository (:name in //pkg becomes //pkg:name, @spec_repo//specs:name is kept) and resolves the label itself as usual. unit and component pass these pairs to both runners in a manifest (TEST_CASE_COVERAGE_SPEC_MANIFEST), one <json path><TAB><reference> per line.

For each test case with a test-spec property the runners normalize the reference (@@//pkg:name, @//pkg:name and //pkg/name become //pkg:name) and look it up:

1. among the references from the test_specs entries;

2. otherwise among the repository-relative labels of the JSON files.

A key that two different specifications claim is ambiguous and does not resolve. A reference that does not resolve is reported like a missing Given/Then and is subject to the same maturity handling.

None of this depends on canonical repository names, so a SEooC that is built as an external dependency (e.g. by an integrator module) resolves its references the same way.