File based rule checks#

Test Function#

The functionality of the Sphinx build rules can be verified with test rst files.

The function test_rst_files in test_rules_file_based.py is executed for each rst file in the directory rst. It creates a SphinxTestApp and a document source folder with an index.rst file that contains a toctree with the given rst file.

It uses the SphinxTestApp to build the documentation and checks for the expected/not expected warnings.

It has it’s OWN conf.py, so if changes need to be made ensure they are made in there

Create a test rst file#

To add a new test case either create a new rst file in the rst directory, or add it to an existing one if it tests something similar The test files can also be organized in a subfolder structure below directory rst.

Each test file consists of two parts:

  1. Exactly one test_metadata need at the top of the file. It declares which check(s) to run and links the test to the requirements it verifies.

  2. One or more Sphinx-Needs directives. A need that should (or should not) emit a warning carries an :expect: / :expect_not: option describing that warning.

The test_metadata need#

test_metadata is now a real Sphinx-Needs directive (not a comment). There must be exactly one per file:

.. code-block:: rst

.. test_metadata::
   :id: test_metadata__<unique_name>
   :partially_verifies_list: <requirement id(s)>
   :test_type: requirements_based
   :derivation_technique: requirements_based

   <description of what this file tests>

<check functions> All checks are run in each rst test file to ensure no contradiction and make testing easier / more complete

<requirement id(s)>:partially_verifies_list: / :fully_verifies_list:
The requirement(s) this test verifies. At least one of :partially_verifies_list: or :fully_verifies_list: must be provided, otherwise the test fails.

:test_type: / :derivation_technique:
Describe how the test was derived. They are attached as properties to the test result.

Needs with expectations#

The :expect: / :expect_not: options live on the need that is being tested. There is no separate statement and no [+x] line offset anymore — the warning is matched against the need’s own location.

<warning message> — :expect: / :expect_not:
Message text which is expected / not expected during the Sphinx build for this need. You only need to match a unique substring, not the full message.

A need without :expect: or :expect_not: is treated as a setup / prerequisite need (for example a link target) and is not asserted against.

Example:

.. code-block:: rst

.. test_metadata::
   :id: test_metadata__mandatory_options
   :partially_verifies_list: tool_req__docs__status
   :test_type: requirements_based
   :derivation_technique: requirements_based

   Tests that mandatory options are enforced.

.. std_wp:: Test requirement
   :id: std_wp__test__abcd
   :expect: std_wp__test__abcd: is missing required attribute: `status`.

This example verifies that the warning message std_wp__test__abcd: is missing required attribute: `status` is shown during the Sphinx build. Only the check_options check is enabled.

Expect-not example:

.. std_wp:: Test requirement
   :id: std_wp__test_options__abce
   :status: active
   :expect_not: attribute

Here the need is fully valid, so we assert that no warning containing the substring attribute is emitted for it.

Note: RST comments used to separate or describe cases (for example .. All required options are present) must not contain ::, otherwise the parser counts them as needs.

Graph check tests#

Graph checks (defined in metamodel.yaml under graph_checks:) are all executed by a single Python function check_metamodel_graph. Use that function name in the test_metadata need’s :check: option:

.. code-block:: rst

.. test_metadata::
   :id: test_metadata__graph_<name>
   :partially_verifies_list: <requirement id>
   :test_type: requirements_based
   :derivation_technique: requirements_based

   <description>

Graph check test files live in tests/rst/graph/.

Warning message format#

You only need to match a unique substring, not the full message. For example:

:expect: wp__bad: Parent need `std_req__aspice_40__bp`

.. hint::

For :expect: use as much as possible of the error message, however for :expect_not: use as little as possible to catch all other issues that MIGHT occur and to avoid for typos etc.

Setup needs#

Prerequisite needs (link targets, shared fixtures) carry no :expect: / :expect_not: option. The framework only asserts on needs that have one of these options, so unannotated needs are invisible to the assertion logic.

The “exempt” case#

A need that does not meet the condition: in the YAML check definition is not selected by the check at all — no warning is emitted. Test this with :expect_not::

.. code-block:: rst

.. workproduct:: No link at all
   :id: wp__no_link
   :expect_not: <something from the explanation>

Full example (graph check)#

.. code-block:: rst

.. test_metadata::
   :id: test_metadata__graph_aspice_40
   :partially_verifies_list: gd_req__aspice_40__parent_link
   :test_type: requirements_based
   :derivation_technique: requirements_based

   Verifies the ASPICE 40 IIC parent-link graph check.

.. std_req:: ASPICE 40 IIC requirement
   :id: std_req__aspice_40__iic_1
   :status: valid

.. std_req:: ASPICE 40 non-IIC requirement
   :id: std_req__aspice_40__bp_1
   :status: valid

.. Positive: links to allowed target — no warning.

.. workproduct:: Valid workproduct
   :id: wp__valid
   :complies: std_req__aspice_40__iic_1
   :expect_not: ASPICE 40 IIC

.. Exempt: no complies link — condition not met, check skipped.

.. workproduct:: Workproduct without link
   :id: wp__no_link
   :expect_not: ASPICE 40 IIC

.. Negative: links to disallowed target — warning expected.

.. workproduct:: Invalid workproduct
   :id: wp__bad
   :complies: std_req__aspice_40__bp_1
   :expect: wp__bad: Parent need `std_req__aspice_40__bp_1`