Design Decisions#

In-Process Health Monitor#

In-Process Health Monitor
status: accepted
consequences: Inter process communication is only required for the alive monitoring towards the Launch Manager and the configuration becomes simpler, at the cost of a harder safety argumentation.
context: Architecture
decision: The Health Monitor is realized as a library linked into the monitored application.
version: 1

The Health Monitor is realized as a library linked into the monitored application rather than as a separate monitoring process. The motivation and the resulting trade-offs are the following.

Context#

The Health Monitor provides the application health monitoring functions (checkpoint supervision, deadline supervision and logical supervision) in a similar fashion as the AUTOSAR Platform Health Manager (PHM). These functions can either be linked into the monitored application or be placed in a separate monitoring process.

Consequences#

  • Inter process communication (IPC) is only needed for the alive monitoring between Health Monitor and Launch Manager

  • Easier configuration
    • The monitoring rules can be configured dynamically on demand

    • The monitoring can be started and stopped dynamically

  • Debugging of problems possibly easier
    • Mapping of the events from a single process vs. the monitored application and the monitor

Alternatives Considered#

Classical external process monitoring#

In a classical external process monitoring, the application process only reports checkpoints via IPC to a separate monitoring daemon. The monitoring daemon is configured with the supervisions that shall be checked on the reported checkpoints.

In this approach the safety argumentation is clear: The monitoring is completely done in a separate process.

Drawbacks of the selected solution over classical external process monitoring:

  • Harder safety argumentation. The following chapter describes the issues and the possible solutions.

Justification for the decision#

As the Health Monitor is linked as part of the monitored application, it raises the following concerns with respect to safety:

  • How can it be ensured that the monitored application does not interfere with the monitoring functionality?

  • How can it be ensured that the Health Monitor does not incorrectly report alive to the Launch Manager when it has detected a supervision error?

These concerns are valid, but can be addressed using the following techniques:

  • Hiding of the internal data from the user:
    • For example, if the monitoring is implemented in a thread, the thread ID must not be exposed to the calling application.

    • Hide the implementation for example with the pImpl-approach

  • Protecting the memory of the library by using guard pages where the application memory is located, and protecting it with mprotect()
    • possibly with the help of a custom allocator

  • Protecting the data with a checksum and possibly a sequence counter
    • The internal data of the library can be checksum’ed every operation cycle, and by adding for example, a sequence counter (or some more complex mathematical function), further checkpoints for detecting misbehavior can be implemented

  • Using a safe programming language, which does not allow a raw pointer access

  • Testing the application with Valgrind etc.

  • Redundant monitoring, if the self monitoring suggested before is not sufficient, a separate library can be implemented in a different memory location to detect sporadic corruption of the Health Monitor’s memory location.

Launch Manager starts processes on a thread pool#

Thread Pool for process starting and stopping
status: accepted
consequences: Process lifecycle operations are organized around a configurable thread pool
context: Architecture
decision: The Launch Manager uses a configurable thread pool to start and stop processes concurrently
version: 1

The Launch Manager uses a configurable thread pool to start and stop processes concurrently. Projects can adjust the pool size to meet their needs and reduce startup time.

Context#

The Launch Manager is responsible for starting and stopping system processes. Large systems may require many processes to be started or stopped, with multiple operations able to run concurrently. Fast startup is an important performance goal for the Launch Manager.

Consequences#

The Launch Manager’s process lifecycle operations are organized around a configurable thread pool, whose size can be adapted to project requirements.

Launch Manager Alive Monitoring is polling-based#

Launch Manager Alive Monitoring is polling-based
status: accepted
consequences: Reduced CPU load, with failure detection delayed by up to one polling cycle
context: Architecture
decision: The Launch Manager evaluates alive notifications periodically rather than as they arrive
version: 1

The Launch Manager polls IPC channels at a configurable interval, collecting alive notifications and evaluating alive supervisions.

Applications write notifications to an IPC buffer without waiting for the Launch Manager to process them. This reduces the Launch Manager’s CPU load compared with handling each notification as it arrives.

Context#

The Launch Manager supervises application processes by checking whether they send alive notifications at the expected frequency. A system may have many application processes sending notifications at short intervals.

Consequences#

Because the Launch Manager evaluates notifications once per cycle instead of as they arrive, it uses less CPU to process them. An alive supervision failure may therefore go undetected until the next polling cycle. Configure the cycle time to balance the desired detection time against acceptable CPU load.

Flatbuffer configuration#

Flatbuffer Configuration
status: accepted
consequences: Fast configuration loading
context: Architecture
decision: The Launch Manager uses a binary configuration format to reduce startup time
version: 1

The Launch Manager uses FlatBuffers as its binary configuration format to reduce startup time.

Context#

At startup, the Launch Manager reads configuration files to determine which processes to start. Loading this configuration is on the time-critical startup path.

Consequences#

The Launch Manager configuration must first be compiled into a binary format for the target. Compared with a textual format such as JSON, the FlatBuffers binary configuration can be loaded faster, particularly for larger configuration files.

Kill process after supervision failure#

Kill process after supervision failure
status: accepted
consequences: A process that failed supervision stops immediately, without a chance to clean up
context: Architecture
decision: The Launch Manager terminates a process with SIGKILL when its alive supervision fails
version: 1

Context#

The Launch Manager can start processes with alive supervision. A supervised process must send alive notifications at the configured frequency. If it does not, its alive supervision fails and the Launch Manager initiates a recovery action.

Unlike a crashed process, a process that fails alive supervision is still running. Its state is unknown: it may be hanging, stuck in a loop, or otherwise malfunctioning. It keeps running until the Launch Manager terminates it.

Options Considered#

Option A: Send SIGTERM, then SIGKILL after the shutdown timeout#

Pros:

  • Consistent with the regular shutdown of a process.

  • The process can clean up before it exits.

Cons:

  • In the worst case, the process keeps running in an unknown and possibly unsafe state for the full shutdown timeout before it is killed.

  • A process that failed supervision may not be able to handle SIGTERM correctly, so the timeout is likely to expire anyway.

Option B: Send SIGKILL immediately#

Pros:

  • The process stops as soon as the supervision failure is detected and cannot continue running in an unsafe state.

  • Recovery can start without waiting for the shutdown timeout.

Cons:

  • The process cannot clean up, for example by persisting data or closing communication channels.

Decision#

Option B: Send SIGKILL immediately.

A process that failed supervision cannot be trusted to shut down correctly. Stopping it immediately is safer than letting it run for the duration of the shutdown timeout.

Consequences#

The Launch Manager kills a process as soon as it detects a supervision failure. Applications must not rely on a graceful shutdown after a supervision failure. Data that has to survive such a failure must be persisted during normal operation.