Test Specification Guideline#

Test specification#

A test specification defines a unique test case in a structured format. Its properties identify the applicable test type, verification level, implementation conditions, and the requirements or architectural elements to which the test case is traced.

A test case may be executed multiple times for different platform variants, platform versions, programming languages, environments, or configurations. These execution-specific conditions are defined by the applicable test scenario, while the unique test case ID remains unchanged. This allows the same test case to be reused across different executions while maintaining consistent traceability to the elements it verifies.

Each test execution produces a specific result for the selected test scenario and version of the software under test. Module and platform verification reports therefore summarize the results of specific execution runs, whereas the unique test case ID provides traceability between the test specification and its results across different runs, variants, and versions.

The test specification shall contain sufficient information to understand and reproduce the intended test. This includes the test objective, required inputs and their relevant ranges, expected results, test environment (e.g. target hardware or simulation and operating system), and, where applicable, expected timing, sequencing, and performance behavior.

A test specification contains the following attributes.

Table 58 Test specification properties#

Property

Type / Values

Description

Helpful links

Partially and/or Fully Verifies

Docs-as-code Id(s) for traceable elements

Links to one or more requirement Ids. Can be partial or full coverage of a requirement, as per test property.

Description

Text

The description should include

  • The objective of the test.

  • Inputs

  • Expected outcome/output/result (e.g. “A success message is displayed.” or “Result should be 42.”)

  • Test environment (e.g. network configuration, clean system state)

  • Expected time sequence of events and behavior (where applicable and where such an expectation exists)

TestType

Examples are:

  • requirements-based

  • interface-test

These are example values and an incomplete list. A full list of test types is available in Verification Concept at Verification Methods.

DerivationTechnique

Examples are:

  • requirements-analysis

  • boundary-values

  • equivalence-classes

  • error-guessing

These are example values and an incomplete list. A full list of test methods is available in Verification Concept at Verification Methods.

The implementation of Verification Plan defines the full list of allowed types and methods.

It is assumed that tests will be written as code (also for manual tests, which are script-based) and that each test case will have a unique identifier (e.g. its script name, execution call, or function name). The invocation used to execute the test defines the test case identification, thereby supporting proper traceability and reproducibility.

As the tests are stored in a repository close to the implementation code, versioning is handled by the repository’s version control.

Any specification and resulting implementation ends with a clear pass/fail result.

Test description#

The test description shall contain sufficient information to understand and reproduce the intended test. It also requires sufficient detail to understand how the linked upstream requirement or architectural element is verified. Examples of this are the test objective, required inputs and their relevant ranges, expected results, test environment (e.g. target hardware or simulation and operating system), and, where applicable, expected timing, sequencing, and performance behavior.

Following Given-When-Then is a preferred way to write test specifications.

  • Given (alternatively // Setup )
    This section contains the setup of the input.
  • Expected
    This Section contains the expectation for calls to mocks or stubs
  • When (alternatively // Call )
    This section contains the call to the software under test.
  • Then (alternatively // Expect )
    This section contains the verification criteria of the test (e.g. all EXPECT statements).
TEST(CreateObjectHistory, GivenVehicleInFront_ExpectHistoryBehindFrontVehicle)
{
   // Given
   const auto whatever_input_values = CreateVehicleInFront();

   // Expected
   EXPECT_CALL(..., ...);

   // When
   const auto history = CreateObjectHistory(whatever_input_values, ...);

   // Then
   for (const auto point : history)
   {
      EXPECT_EQ(..., ...);
   }
}

Another example for a good test description could be:

Verify successful login with valid credentials.
Input:
- Username: testuser
- Password: password123
Expected Outcome:
- User is redirected to the home page and the welcome message “Hello, testuser!” is displayed.

A example for a bad test description could be:

Test user login

Additional information can also be found in Verification Guideline.

The specification is part of the test implementation and has to comply to the requirements specified in Linking Requirements to Tests.