Tool Qualification Report Example

This page demonstrates the machine-readable model used for a Docs-as-Code tool qualification report. The content is based on the published S-CORE Doc-as-Code Tool Verification Report.

The evaluation considers build and CI behavior, pull-request review, and derived views. A malfunction is represented as a nested Need below the use case where it occurs. Its safety_measure is intentionally free text. Generated testcase Needs can link back to a malfunction through the existing verification links.

Docs-as-Code Tool Qualification		status: evaluated
safety_affected: YES security_affected: YES tcl: LOW tool_version: v7.0.1 version: 2 realizes: wp__tool_verification_report
This report evaluates the Docs-as-Code toolchain as a qualified tool for producing and verifying safety-relevant documentation.
doc_tool__docs_as_code		Tool Verification Report

Automatically generated overview starts here.

Tool qualification matrix

The following table summarizes the tool use cases and their identified malfunctions. Each malfunction is nested below its use case and links to the requirements it violates.

Table 1 Tool qualification matrix

Malfunction identification	Use case description	Malfunction	Impact on safety?	Safety measure available?	Detection sufficient?	Additional safety measure required?	Confidence
Listing assumptions of use (tool_malfunction__docs_as_code__aou_listing)	Build and CI behavior	A missing or incorrect assumption of use can lead to an incomplete safety argument.	YES	YES	NO	YES	LOW
Requirements coverage stati... (tool_malfunction__docs_as_code__coverage)	Build and CI behavior	A defect in the statistics calculation can produce an incorrect coverage result while leaving the build green.	YES	YES	NO	YES	LOW
Safety-critical linking enf... (tool_malfunction__docs_as_code__safety_links)	Build and CI behavior	A silent false negative could allow an unsafe derivation or other invalid safety-critical link to pass the build.	YES	YES	NO	YES	LOW
Document metamodel enforcement (tool_malfunction__docs_as_code__metamodel)	Build and CI behavior	A silent false negative can result from a permissive regular expression or a defect in a metamodel check.	YES	YES	NO	YES	LOW
Test linkage (tool_malfunction__docs_as_code__test_linkage)	Build and CI behavior	The safety case may incorrectly conclude that a requirement is tested if a testcase link is missing or points to the wrong need.	YES	YES	NO	YES	LOW
Test reference check (tool_malfunction__docs_as_code__test_refs)	Build and CI behavior	A missing or outdated requirement reference can invalidate the test evidence without causing a CI failure.	YES	YES	NO	YES	LOW
Architecture visualization (tool_malfunction__docs_as_code__arch_view)	Derived views	An incorrect visualization can misrepresent the architecture. The result is detected by inspecting the derived view.	YES	YES	YES	NO	HIGH
Backlinks (tool_malfunction__docs_as_code__backlinks)	Derived views	A generated backlink can be missing or point to the wrong Need.	NO	YES	NO	YES	LOW
Documentation generation (tool_malfunction__docs_as_code__generation)	Derived views	The generated HTML can be incomplete, outdated, or rendered incorrectly. The issue is detected by inspecting the derived view.	NO	YES	YES	NO	HIGH

Traceability evidence

The evidence below keeps requirement violations, safety measures and generated testcase links visible without hiding them in the summary table.

Table 2 Malfunction traceability

Malfunction	Violates requirements	Safety measure	Fully verified by	Partially verified by
Listing assumptions of use (tool_malfunction__docs_as_code__aou_listing)	Requirement linkage to AoU ... (tool_req__docs_req_link_covers_aou)	Each change to an assumption of use is reviewed by a committer.	—	—
Requirements coverage stati... (tool_malfunction__docs_as_code__coverage)	Provide Metrics for linked ... (tool_req__docs_test_linkage_metrics)	Coverage results are reviewed as part of the tool qualification evidence.	—	testcase__test_traceability_metrics__test_calculate_requirement_metrics_counts_links_and_missing_ids_zjedt, testcase__test_traceability_metrics__test_calculate_test_metrics_counts_linked_tests_and_broken_refs_atrpa
Safety-critical linking enf... (tool_malfunction__docs_as_code__safety_links)	Safety: enforce safe linking (tool_req__docs_common_attr_safety_link_check)	Pull-request review checks changes to safety-critical links before merge.	testcase__test_rules_file_based__test_rst_files[graph/test_metamodel_graph.rst]_cjues	—
Document metamodel enforcement (tool_malfunction__docs_as_code__metamodel)	Document Types (tool_req__docs_doc_types)	Repository contents are the source of truth and every change is reviewed by a committer.	testcase__test_metamodel_load__test_all_undeclared_links_are_reported_at_once_inhvd, testcase__test_metamodel_load__test_shipped_metamodel_declares_every_link_it_uses_uueqj, testcase__test_metamodel_load__test_tool_qualification_types_model_nested_traceability_tdpbb	—
Test linkage (tool_malfunction__docs_as_code__test_linkage)	Supports linking to test cases (tool_req__docs_test_link_testcase)	Test metadata and requirement links are reviewed when the test suite changes.	testcase__test_testlink__test_testcaseneed_to_dict_multiple_links_vstss, testcase__test_xml_parser__test_read_test_xml_file_xofsm	—
Test reference check (tool_malfunction__docs_as_code__test_refs)	Supports linking to test cases (tool_req__docs_test_link_testcase)	Pull-request review checks that referenced requirements remain current.	testcase__test_testlink__test_testcaseneed_to_dict_multiple_links_vstss, testcase__test_xml_parser__test_read_test_xml_file_xofsm	—
Architecture visualization (tool_malfunction__docs_as_code__arch_view)	Support Diagram drawing of ... (tool_req__docs_arch_views)	The generated architecture view is inspected during pull-request review.	testcase__test_rules_file_based__test_rst_files[architecture/architecture_tests.rst]_zxsba	—
Backlinks (tool_malfunction__docs_as_code__backlinks)	Support machine-readable mo... (tool_req__docs_verification_report_need)	Backlinks are covered by the generated documentation checks.	testcase__test_source_code_link_integration__test_source_link_integration_ok_jmkot	—
Documentation generation (tool_malfunction__docs_as_code__generation)	Document Types (tool_req__docs_doc_types)	The generated HTML is inspected as a derived view.	testcase__test_source_code_link_integration__test_source_link_integration_ok_jmkot	—

Build and CI behavior
version: 1 contains: tool_malfunction__docs_as_code__aou_listing, tool_malfunction__docs_as_code__coverage, tool_malfunction__docs_as_code__metamodel, tool_malfunction__docs_as_code__safety_links, tool_malfunction__docs_as_code__test_linkage, tool_malfunction__docs_as_code__test_refs
The repository contents are the source of truth and gated CI evaluates the generated documentation and traceability data. Document metamodel enforcement additional_safety_measure_required: YES detection_sufficient: NO safety_affected: YES safety_measure: Repository contents are the source of truth and every change is reviewed by a committer. version: 1 contained by: tool_usecase__docs_as_code__build_ci violates: tool_req__docs_doc_types A silent false negative can result from a permissive regular expression or a defect in a metamodel check. tool_malfunction__docs_as_code__metamodel Tool Verification Malfunction Safety-critical linking enforcement additional_safety_measure_required: YES detection_sufficient: NO safety_affected: YES safety_measure: Pull-request review checks changes to safety-critical links before merge. version: 1 contained by: tool_usecase__docs_as_code__build_ci violates: tool_req__docs_common_attr_safety_link_check A silent false negative could allow an unsafe derivation or other invalid safety-critical link to pass the build. tool_malfunction__docs_as_code__safety_links Tool Verification Malfunction Requirements coverage statistics additional_safety_measure_required: YES detection_sufficient: NO safety_affected: YES safety_measure: Coverage results are reviewed as part of the tool qualification evidence. version: 1 contained by: tool_usecase__docs_as_code__build_ci violates: tool_req__docs_test_linkage_metrics A defect in the statistics calculation can produce an incorrect coverage result while leaving the build green. tool_malfunction__docs_as_code__coverage Tool Verification Malfunction Test linkage additional_safety_measure_required: YES detection_sufficient: NO safety_affected: YES safety_measure: Test metadata and requirement links are reviewed when the test suite changes. version: 1 contained by: tool_usecase__docs_as_code__build_ci violates: tool_req__docs_test_link_testcase The safety case may incorrectly conclude that a requirement is tested if a testcase link is missing or points to the wrong need. tool_malfunction__docs_as_code__test_linkage Tool Verification Malfunction Test reference check additional_safety_measure_required: YES detection_sufficient: NO safety_affected: YES safety_measure: Pull-request review checks that referenced requirements remain current. version: 1 contained by: tool_usecase__docs_as_code__build_ci violates: tool_req__docs_test_link_testcase A missing or outdated requirement reference can invalidate the test evidence without causing a CI failure. tool_malfunction__docs_as_code__test_refs Tool Verification Malfunction Listing assumptions of use additional_safety_measure_required: YES detection_sufficient: NO safety_affected: YES safety_measure: Each change to an assumption of use is reviewed by a committer. version: 1 contained by: tool_usecase__docs_as_code__build_ci violates: tool_req__docs_req_link_covers_aou A missing or incorrect assumption of use can lead to an incomplete safety argument. tool_malfunction__docs_as_code__aou_listing Tool Verification Malfunction
tool_usecase__docs_as_code__build_ci		Tool Verification Use Case

Derived views
version: 1 contains: tool_malfunction__docs_as_code__arch_view, tool_malfunction__docs_as_code__backlinks, tool_malfunction__docs_as_code__generation
Derived views are generated from the machine-readable Need model and are not the source of truth. Architecture visualization additional_safety_measure_required: NO detection_sufficient: YES safety_affected: YES safety_measure: The generated architecture view is inspected during pull-request review. version: 1 contained by: tool_usecase__docs_as_code__derived_views violates: tool_req__docs_arch_views An incorrect visualization can misrepresent the architecture. The result is detected by inspecting the derived view. tool_malfunction__docs_as_code__arch_view Tool Verification Malfunction Backlinks additional_safety_measure_required: YES detection_sufficient: NO safety_affected: NO safety_measure: Backlinks are covered by the generated documentation checks. version: 1 contained by: tool_usecase__docs_as_code__derived_views violates: tool_req__docs_verification_report_need A generated backlink can be missing or point to the wrong Need. tool_malfunction__docs_as_code__backlinks Tool Verification Malfunction Documentation generation additional_safety_measure_required: NO detection_sufficient: YES safety_affected: NO safety_measure: The generated HTML is inspected as a derived view. version: 1 contained by: tool_usecase__docs_as_code__derived_views violates: tool_req__docs_doc_types The generated HTML can be incomplete, outdated, or rendered incorrectly. The issue is detected by inspecting the derived view. tool_malfunction__docs_as_code__generation Tool Verification Malfunction
tool_usecase__docs_as_code__derived_views		Tool Verification Use Case