Design Decisions#
In-Process Health Monitor#
In-Process Health Monitor
|
status: accepted
|
||||
The Health Monitor is realized as a library linked into the monitored application rather than as a separate monitoring process. The motivation and the resulting trade-offs are the following.
Context#
The Health Monitor provides the application health monitoring functions (checkpoint supervision, deadline supervision and logical supervision) in a similar fashion as the AUTOSAR Platform Health Manager (PHM). These functions can either be linked into the monitored application or be placed in a separate monitoring process.
Consequences#
Inter process communication (IPC) is only needed for the alive monitoring between Health Monitor and Launch Manager
- Easier configuration
The monitoring rules can be configured dynamically on demand
The monitoring can be started and stopped dynamically
- Debugging of problems possibly easier
Mapping of the events from a single process vs. the monitored application and the monitor
Alternatives Considered#
Classical external process monitoring#
In a classical external process monitoring, the application process only reports checkpoints via IPC to a separate monitoring daemon. The monitoring daemon is configured with the supervisions that shall be checked on the reported checkpoints.
In this approach the safety argumentation is clear: The monitoring is completely done in a separate process.
Drawbacks of the selected solution over classical external process monitoring:
Harder safety argumentation. The following chapter describes the issues and the possible solutions.
Justification for the decision#
As the Health Monitor is linked as part of the monitored application, it raises the following concerns with respect to safety:
How can it be ensured, that the monitored application does not interfere with the monitoring functionality?
How can it be ensured, that the Health Monitor does not incorrectly report alive to the Launch Manager when it has detected a supervision error?
These concerns are valid, but can be addressed using the following techniques:
- Hiding of the internal data from the user:
For example, if the monitoring is implemented in a thread, the thread ID must not be exposed to the calling application.
Hide the implementation for example with the pImpl-approach
- Protecting the memory of the library by using guard pages where the application memory is located, and protect it with mprotect()
possibly with a help of a custom allocator
- Protecting the data with a checksum and possibly a sequence counter
The internal data of the library can be checksum’ed every operation cycle, and by adding for example, a sequence counter (or some more complex mathematical function), further checkpoints for detecting misbehavior can be implemented
Using a safe programming language, which does not allow a raw pointer access
Testing the application with Valgrid etc.
Redundant monitoring, if the self monitoring suggested before is not sufficient, a separate library can be implemented in a different memory location to detect sporadic corruption of the Health Monitors memory location.
Launch Manager starts processes on a thread pool#
Thread Pool for process starting and stopping
|
status: accepted
|
||||
The Launch Manager uses a configurable thread pool to start and stop processes concurrently. Projects can adjust the pool size to meet their needs and reduce startup time.
Context#
The Launch Manager is responsible for starting and stopping system processes. Large systems may require many processes to be started or stopped, with multiple operations able to run concurrently. Fast startup is an important performance goal for the Launch Manager.
Consequences#
The Launch Manager’s process lifecycle operations are organized around a configurable thread pool, whose size can be adapted to project requirements.
Launch Manager Alive Monitoring is polling-based#
Launch Manager Alive Monitoring is polling-based
|
status: accepted
|
||||
The Launch Manager polls IPC channels at a configurable interval, collecting alive notifications and evaluating alive supervisions.
Applications write notifications to an IPC buffer without waiting for the Launch Manager to process them. This reduces the Launch Manager’s CPU load compared with handling each notification as it arrives.
Context#
The Launch Manager supervises application processes by checking whether they send alive notifications at the expected frequency. A system may have many application processes sending notifications at short intervals.
Consequences#
Because the Launch Manager evaluates notifications once per cycle instead of as they arrive, it uses less CPU to process them. An alive supervision failure may therefore go undetected until the next polling cycle. Configure the cycle time to balance the desired detection time against acceptable CPU load.
Flatbuffer configuration#
Flatbuffer Configuration
|
status: accepted
|
||||
The Launch Manager uses FlatBuffers as its binary configuration format to reduce startup time.
Context#
At startup, the Launch Manager reads configuration files to determine which processes to start. Loading this configuration is on the time-critical startup path.
Consequences#
The Launch Manager configuration must first be compiled into a binary format for the target. Compared with a textual format such as JSON, the FlatBuffers binary configuration can be loaded faster, particularly for larger configuration files.