IPC Design Decisions#

Selection of LoLa Message Passing as IPC#

Selection of LoLa Message Passing as IPC		status: proposed
context: doc__crypto_architecture decision: Use LoLa Message Passing, implemented with the score::message_passing API, as the crypto daemon control-plane IPC transport. version: 1

dec_rec__crypto__lola_message_passing_ipc		Decision Record

LoLa Message Passing, implemented with the score::message_passing API, is selected as the target control-plane transport for communication between client processes and the daemon. Production integration and implementation validation are follow-up work.

Context#

The IPC transport must be suitable for use in an ASIL-B environment. This includes deterministic and bounded execution and resource behavior, controlled heap/allocation use on ASIL-relevant paths, suitable isolation for the deployment, and appropriate quality artifacts such as requirements and design traceability, analysis, verification, and compliance or qualification evidence. It must also provide bounded client-side waiting, allow concurrent calls from multiple client processes and threads, expose the identity of the peer, avoid imposing the daemon’s worker-thread model, and keep the application protocol independent of operating-system transport details.

Decision#

Use score::message_passing with FlatBuffers as the serialized control-plane payload format. The application uses the ClientFactory and ServerFactory abstractions and does not depend directly on Unix-domain socket or QNX message-passing APIs.

The deployment configuration shall establish fixed system-level upper bounds:

N: maximum number of client processes and server connections.
T: maximum number of concurrent in-flight requests per client process.
M: maximum serialized payload size in bytes.

The client and server must use consistent protocol and queue-size settings. The relevant capacities shall be derived from N, T, and M, and the expected server connections shall be pre-allocated. These values are integration parameters, not runtime-adaptive values, and must be reviewed when the deployment topology or concurrency assumptions change.

Consequences#

Positive:

Kernel-mediated transport provides process isolation without a shared writable memory region between client and daemon processes.
The server can retrieve kernel-provided peer credentials from an active connection, avoiding reliance on a client-supplied identity.
The application-facing API is OS-agnostic; the framework selects the native backend for the target operating system.
The daemon retains control of its execution model and can dispatch work to a fixed worker pool with appropriate implementation.
The communication module documents safety-oriented quality tooling and ASIL-B qualification, while the LoLa Message Passing design supports fixed resource bounds, preallocation, and pool/monotonic allocation.
Multiple client processes and multiple concurrent client threads are supported. A shared connection per client process avoids the client-process-times-thread connection growth of a per-thread design.
A service identifier is sufficient to locate the endpoint; no per-client service configuration or service registry is required by the application.

Negative:

Safe operation requires explicit worst-case sizing for N, T, and M. In particular, an undersized QNX notify queue can cause ENOBUFS and a lost response, while inconsistent payload sizes can cause send or receive failures.
The selected LoLa Message Passing implementation does not provide inter-VM communication. Inter-VM deployment depends on a future framework backend. The separate connection implementation suggested by the crypto IPC abstraction is only a possible migration direction.
The transport is not shared-memory based, so it may have higher per-message overhead than LoLa for very high-throughput data transfer. This is not a control-plane requirement because bulk data is handled separately.

Alternatives Considered#

gRPC over Unix-Domain Socket#

gRPC provides mature request/response and inter-VM channel options, but the general-purpose library uses framework-managed threads and dynamic resources and does not provide an ASIL-oriented deterministic resource profile or safety qualification artifacts. The evaluated blocking call also does not establish the required bounded waiting behaviour. It remains suitable for QM-to-QM communication and is the strongest option for an immediate network or inter-VM channel.

LoLa Full SOA Abstraction#

LoLa is documented as safety-oriented/ASIL-B qualified, with custom memory-management infrastructure and quality tooling. The synchronous method lacks an application escape from an indefinitely blocked call. The asynchronous method layer built on top during POCs, improves server threading and response timeout handling, but its internal method call still blocks without a timeout and its broadcast response model requires additional per-client skeleton instances and response routing.

LoLa Message Passing#

The message-passing abstraction exposes platform-independent client and server factories while keeping the operating-system transport behind the IPC layer. Using SendWithCallback with Reply and Notify separates request acceptance from operation completion. A shared connection per client process supports concurrent client threads through explicit request identifiers and a server-side worker pool, without scaling connections as N * T. The design requires explicit bounds for client processes, in-flight requests, payload size, and transport queues, but preserves process isolation and the daemon’s freedom to choose its worker model.

Comparison Summary#

The alternatives were assessed against ASIL-B suitability, bounded waiting, connection-loss handling, authentic peer identification, configuration effort, server-threading flexibility, concurrency, one-to-one communication, latency, and inter-VM support.

gRPC over Unix-domain sockets provides mature request/reply semantics and the strongest network and inter-VM path. Its general-purpose runtime does not provide the deterministic resource profile or safety evidence required for the intended ASIL-B use. The current adapter also has no configured deadline, and peer authentication would require an additional security mechanism.
LoLa Full SOA synchronous Method provides a safety-oriented service model and one-to-one responses, but its blocking call offers no application escape when the server stalls. Its service configuration is also more extensive than the selected solution requires.
LoLa Full SOA asynchronous Method plus Event allows independent server worker execution and bounded waiting for the final event. The initial method call remains blocking without a framework change, and broadcast event delivery requires additional per-client instances and response routing.
LoLa Message Passing provides platform-independent client and server factories, process-isolating transport, kernel-provided peer credentials, and a server-controlled worker model can be added easily. SendWithCallback followed by Reply and Notify supports concurrent calls over one connection per client process, with application-level request identifiers for response routing. It requires explicit resource bounds and a typed timeout and connection-loss contract. It does not provide inter-VM communication.

LoLa Message Passing is selected because it best satisfies the ASIL-B, bounded-client-waiting, peer-identity, concurrency, and server-threading goals while keeping operating-system transport details below the application IPC interface. The remaining backend, error-contract, resource-sizing, and inter-VM limitations are explicit implementation and verification work.

Justification for the Decision#

LoLa Message Passing is the only evaluated option that appears capable of meeting the broader ASIL-B use goals and the bounded client call-path goal based on its library design and documented quality properties. Meeting the timeout and connection-loss requirements still depends on the production IPC wrapper, and a typed error contract. The lack of inter-VM communication is accepted, the control-plane IPC abstraction can potentially be used to offer a separate inter-VM transport in the future.

Two-Phase Request/Reply Acknowledgement and Notify Completion#

Two-Phase Request/Reply Acknowledgement and Notify Completion		status: proposed
context: doc__crypto_architecture decision: Use SendWithCallback for a non-blocking request acknowledgement and Notify for the independently produced operation response. version: 1

dec_rec__crypto__two_phase_ipc_interaction		Decision Record

The selected IPC protocol uses two phases: an immediate Reply acknowledgement for the accepted request, followed by a Notify message containing the operation result.

Context#

Using LoLa Message Passing, the REQUEST/REPLY protocol serializes requests on one connection: the server does not process the next request on that connection until the current request has received a Reply. A client process may nevertheless have many threads issuing independent operations, and the server must be able to process those operations concurrently.

Using a single long-running request callback would keep the shared connection occupied for the duration of the cryptographic operation. Using SendWaitReply would additionally block the client thread inside the IPC library and does not provide the application with a timeout escape if the server stalls. The selected communication model therefore separates request acceptance from operation completion.

Decision#

The protocol is defined as follows:

The client assigns a unique non-zero request_id, inserts a pending-call record, and calls SendWithCallback with the FlatBuffer ControlRequest.
The server validates the request and copies the complete request into the application work queue.
The server calls Reply with a minimal acknowledgement carrying the request_id. After the acknowledgement attempt, it wakes a worker. A successful acknowledgement means the request was admitted to the work queue.
A server worker performs the operation and calls Notify on the same IServerConnection with a ControlResponse carrying the original request_id and the result payload.
The client NotifyCallback routes the response by request_id and signals the waiting application thread. The application waits with a bounded timeout and retires the pending-call record after completion or timeout.

The acknowledgement is an acceptance signal, not the operation result. The transport does not automatically retry requests. An accepted request is attempted once; if the operation result is not observed because of a timeout or delivery failure, the outcome is unknown to the caller. The application may explicitly retry when the operation semantics allow it. ReplyCallback is used to observe send or acknowledgement failure; the client does not complete the operation merely because the acknowledgement arrived.

Consequences#

Positive:

The server releases the per-connection REQUEST/REPLY serialization window before doing the potentially slow operation. Subsequent requests can be accepted while earlier requests execute in the worker pool.
SendWithCallback returns without holding the caller inside the IPC operation. A production wrapper can use a mandatory application-level wait_for to provide a bounded wait for the final response and report a typed timeout.
One connection can be shared by all threads in a client process. The application-level request_id protocol provides deterministic response demultiplexing without requiring one connection per thread.
Notify is point-to-point, so a response is delivered only to the client connection that issued the request. This avoids the cross-client response leakage risk of a broadcast event model.
The server can use a fixed worker pool and must protect connection lifetime while workers complete delayed notifications.

Negative:

The protocol and implementation are more complex than a single synchronous call. Pending-call state, request identifiers, response parsing, timeout cleanup, and late-notification handling are required.
The acknowledgement does not prove that the operation completed; callers must handle both acknowledgement failure and final-notification timeout.
Queue capacities must cover the configured concurrency. Client async-reply and send queues and the server notify queue must be sized consistently. An undersized queue can reject a send or drop a notification.
A worker may finish after the client has timed out. The server must detect a disconnected connection or safely skip the notification, and the client must discard late notifications for retired request identifiers.
Notify can still be transport-dependent in its blocking behaviour. Bounded notification semantics require support from the transport or an additional connection lease abstraction.

Alternatives Considered#

Single SendWaitReply Call#

The client sends a request and waits for the server to return the final result through the REQUEST/REPLY exchange. This is simpler and naturally matches one request to one response, but it blocks the caller inside the IPC library for the whole operation and provides no application-level timeout. With a shared connection, the server callback also serializes all requests until work is complete.

Long-Running SendWithCallback Reply#

The server could defer Reply until the worker has finished and return the final response through ReplyCallback. This preserves a single response channel, but retains the per-connection serialization window for the entire operation. It prevents the shared-connection design from accepting concurrent requests at the intended rate and makes the server callback lifetime depend on operation duration.

LoLa Method plus Broadcast Event#

The LoLa asynchronous method plus event uses a short method call followed by an event, which has similar separation of acceptance and completion. Its event is broadcast, however, so the design needs one skeleton instance per client and additional routing to prevent clients from observing one another’s responses. The Notify primitive provides the same asynchronous completion model as a point-to-point message on the existing client connection.

Justification for the Decision#

Immediate acknowledgement is the smallest operation that satisfies the low-level protocol’s serialization rule while allowing the server to dispatch work independently. The separate Notify completion keeps the client thread out of the IPC library’s blocking path, enables an explicit application timeout, and preserves one connection per client process. The request ID and pending-call lifecycle are deliberate complexity: they are required to obtain bounded, concurrent operation completion from a shared connection without using broadcast communication or shared writable memory.

Reference#

DAR — IPC Mechanism for score-crypto Daemon#


Status	Draft
Date	2026-08-13
Author	ETAS
Context	score-crypto daemon IPC transport selection

1. Problem Statement#

The score-crypto daemon requires an IPC transport to serve cryptographic operations to client processes. The current implementation uses gRPC. The target deployment environment is automotive ECUs with strict functional-safety requirements (ISO 26262), potentially running mixed ASIL-level workloads, possibly across VM boundaries. A replacement or continuation decision is needed.

2. Requirements#

ID	Requirement	Rationale
R1	Suitability for ASIL-B use	The library must be designed and documented to support deterministic and bounded execution and resource behavior, controlled heap/allocation use on ASIL-relevant paths, suitable isolation for the deployment, and appropriate quality artifacts such as requirements/design traceability, analysis, verification, and compliance or qualification evidence.
R2	Bounded timeout on every call	WCET must be provable. A stalled QM server must never indefinitely block an ASIL-B client thread.
R3	Resilience to server crash / connection loss	If the server process dies or the connection is lost, the client must receive a typed error. The client must not block indefinitely or silently operate on a dead connection.
R4	Authentic peer identification	Server must be able to identify the calling process via an OS-enforced mechanism, not a client-supplied value.
R5	Minimal or zero configuration	The IPC usage should require minimal configuration and be fully configurable programmatically.
R6	Flexible server-side threading model	The IPC transport must not impose a threading model. The server must be free to choose single-thread, fixed pool, or per-request threads.
R7	Concurrent calls from one process	Multiple threads in one client process must be able to issue independent calls simultaneously.
R8	Multiple concurrent client processes	Multiple independent processes must each be able to connect and call the server simultaneously.
R9	One-to-one communication	Each request maps to exactly one response. No broadcast, no fan-out.
R10	Low latency	IPC overhead should be low. Data throughput is not a priority, since crypto provides a separate data plane for transfer of bigger data elements.
R11	Inter-VM communication	The mechanism must either natively support inter-VM communication on a safety hypervisor, or have a minimal and safe migration path to do so. Crypto has an IPC / connection abstraction which allows the usage of different IPC / connection mechanisms.

3. Options Considered#

Option A — gRPC over Unix Domain Socket (current)#

gRPC using FlatBuffers serialisation, connected via unix:// channel. In production use in the repo today via GrpcControlClient / GrpcControlServer.

Option B — LoLa Full SOA Abstraction (synchronous Method)#

S-CORE mw::com LoLa binding, synchronous Method call. Single Execute(IpcBuffer) → IpcBuffer method over shared memory. Evaluated in the synchronous LoLa prototype.

Option C — LoLa Full SOA Abstraction (asynchronous Method + Event)#

S-CORE mw::com LoLa binding, two-phase protocol: Phase 1 — short blocking Method call returns a ticket; Phase 2 — result delivered via a broadcast Event. Evaluated in the asynchronous LoLa prototype.

Option D — LoLa Message Passing Abstraction (SendWithCallback + Notify)#

The platform-independent message-passing abstraction uses the S-CORE score::message_passing API with SendWithCallback / Reply / Notify primitives. The transport backend is OS-specific and provided by the framework: Unix domain socket on Linux, QNX message passing on QNX. The application code is OS-agnostic. Evaluated in the low-level message-passing prototype.

4. Decision Matrix#

Options C and D use a two-phase protocol (enqueue + async response); R2 applies to both phases independently. For Options A and B, which are single-phase, R2 describes a single blocking call.

The matrix describes target library and architecture capabilities, not completed production guarantees. ✅ means the option appears capable of meeting the requirement architecturally; ⚠️ means the result depends on a wrapper, backend, configuration, or follow-up implementation; ❌ means an identified architectural mismatch; and ❓ means it was not evaluated. The POCs provide feasibility evidence for selected points, while the production implementation must validate the remaining requirements.

Requirement	Option A — gRPC	Option B — LoLa Full SOA (sync)	Option C — LoLa Full SOA (async)	Option D — LoLa Message Passing Abstraction
R1 ASIL-B suitability	❌ general-purpose gRPC uses framework-managed threads and dynamic runtime resources; the library does not provide an ASIL-oriented deterministic resource profile or safety qualification artifacts	✅ the LoLa library is documented as safety-oriented/ASIL-B qualified and provides custom memory-management infrastructure suitable for bounded resource use and ASIL-B deployment	✅ the LoLa library is documented as safety-oriented/ASIL-B qualified and provides custom memory-management infrastructure suitable for bounded resource use and ASIL-B deployment, including its method/event model	✅ the message-passing library design supports fixed resource bounds, preallocation, and pool/monotonic allocation; the communication module documents safety-oriented quality tooling and ASIL-B qualification
R2 Bounded timeout	⚠️ current adapter uses a blocking call without a configured deadline; deadline-based or async gRPC was not analyzed	❌ single-phase blocking call — blocks indefinitely if server stalls	⚠️ Phase 2 (response event): `wait_for()` indicated in POC; Phase 1 (enqueue): ❌ `DoCall()` holds caller’s thread inside LoLa framework with no timeout API — requires framework change	⚠️ `SendWithCallback()` is non-blocking by design and can support an application-level timeout; mandatory timeout API and typed error handling remain production work, and server-side `Notify()` bounds are backend-dependent
R3 Server crash / connection loss	❓ not analysed in POC	❓ not analysed in POC	❓ not analysed in POC	⚠️ transport disconnect was observed in the POC; typed propagation to all pending calls and cleanup policy remain production work
R4 Authentic peer identification	❌ no authentication mechanism	⚠️ config specifies which UIDs may use a service instance — non-listed UIDs are rejected at runtime by the framework; no API to actively query the connected UID	⚠️ config specifies which UIDs may use a service instance — non-listed UIDs are rejected at runtime by the framework; no API to actively query the connected UID	✅ `score::message_passing` exposes an API to retrieve kernel-provided peer credentials from an active connection
R5 Minimal config	✅ socket path only	❌ LoLa requires service-oriented configuration for the service, instance, method/event identifiers, application identity, and safety-relevant properties; configuration must be provided consistently to the participating applications and composed with any other `mw::com` services used in the same process	❌ LoLa requires service-oriented configuration for the service, instance, method/event identifiers, application identity, and safety-relevant properties; configuration must be provided consistently to the participating applications and composed with any other `mw::com` services used in the same process	⚠️ service identifier only, but worst-case buffer sizes must be configured explicitly: max client processes (N), max concurrent threads per client (T), and max payload size — all must be set to system-wide upper bounds at compile/startup time
R6 Flexible server threading	⚠️ gRPC owns an internal thread pool; handler is called on gRPC threads. Flexible configuration of that pool may not be sufficient for the required threading model, while a complete replacement of it may be feasible but was not evaluated	⚠️ skeleton thread processes one call at a time and blocks until the handler returns; handler may dispatch to a pool internally but must block the skeleton on the result — skeleton is always occupied during work	✅ flexible threading model can be built on top — POC demonstrates a server-side thread pool receiving work via the Phase 1 callback and completing it independently	✅ `Reply()` callable from any thread at any time — server fully controls threading
R7 Concurrent calls / same process	✅ channel is thread-safe, no external mechanism needed	⚠️ concurrent calls depend on the LoLa proxy and method queue configuration; the service abstraction does not make the required parallelism transparent to the application	⚠️ concurrent calls depend on the LoLa proxy, method queue, and event subscription configuration; the service abstraction does not make the required parallelism transparent to the application	⚠️ the library supports concurrent use, while request-ID assignment, pending-call tracking, and response multiplexing must be implemented above the library; sufficient resource configuration is also needed
R8 Multiple client processes	✅ works out of the box	⚠️ supported through the service-oriented deployment model, but requires consistent application/service configuration across participating processes — see R5	⚠️ supported through the service-oriented deployment model, but requires consistent application/service configuration across participating processes — see R5	✅ the library supports one server communicating with multiple client processes through independent client/server sessions; sufficient connection and queue resource configuration is needed
R9 One-to-one	✅ each call gets exactly one response	✅ each call gets exactly one response	⚠️ response is a broadcast event — requires one skeleton instance per client to prevent cross-client response leakage	✅ each call gets exactly one response
R10 Latency	❌ indicative POC measurement ~430 µs (full request + response)	❓ not measured	⚠️ indicative POC measurement ~130 µs (lowest measured value, but still significant IPC overhead)	⚠️ indicative POC measurement ~160 µs (second-lowest measured value, but still significant IPC overhead)
R11 Inter-VM	⚠️ gRPC supports network channels, but the current adapter hardcodes Unix-domain endpoints; an endpoint/configuration change and validation of the inter-VM transport and peer-authentication model are required	⚠️ current LoLa binding is SHM-only (single-kernel); the service-oriented architecture could in principle support a network binding without changing the service API, but no such binding exists today	⚠️ current LoLa binding is SHM-only (single-kernel); the service-oriented architecture could in principle support a network binding without changing the service API, but no such binding exists today	❌ current `score::message_passing` backends are local Unix-domain socket and QNX message passing; the library provides no inter-VM transport. A new framework backend would be required.

Note: R1 and R2 are hard safety blockers — any ❌ on these disqualifies an option for safety use regardless of performance on other requirements. Options A, B, and C all carry at least one ❌ on R1 or R2.

POC note: The LoLa POCs generated per-client configuration and used unique application identifiers to exercise multiple clients. These are prototype workarounds and should not be interpreted as the library’s fundamental configuration model.

5. Analysis#

Option B (LoLa Full SOA, synchronous) meets the R1 library-capability assessment: the LoLa library is documented as safety-oriented/ASIL-B qualified and provides custom memory-management infrastructure suitable for bounded resource use. It fails R2 (timeout): there is no two-phase workaround available — the single blocking call holds the caller’s thread until the handler returns with no escape path. The synchronous prototype was only validated in-process (skeleton and proxy on separate threads within a single test binary); cross-process concurrency was not tested. The config burden is high and would compound if LoLa is used elsewhere in the same process.

Option C (LoLa Full SOA, asynchronous) meets the R1 library-capability assessment: the LoLa library is documented as safety-oriented/ASIL-B qualified and provides custom memory-management infrastructure suitable for bounded resource use, including its method/event model. Its indicative POC latency of approximately 130 microseconds is the lowest measured value, but still represents significant IPC overhead and should remain a warning rather than an unqualified pass. It fails partially on R2: Phase 2 timeout is implemented and flexible server threading is achievable, but Phase 1 timeout requires a LoLa framework change — it cannot be fixed in application code. On server crash, Phase 2 never fires and the client hangs until the application-level timeout expires; the crash itself is not detected independently. The broadcast-event response model requires one skeleton instance per client, adding complexity and config overhead. Notably, LoLa’s service-oriented architecture is transport-agnostic by design; a future network binding could enable inter-VM communication without application-level changes — a meaningful long-term advantage that does not resolve the current safety gaps.

Option A (gRPC) fails R1 for the stated ASIL-B use because the general-purpose library uses framework-managed threads and dynamic resources and does not provide an ASIL-oriented deterministic resource profile or safety qualification artifacts. The current adapter also does not configure a deadline; deadline-based and asynchronous gRPC were not analyzed. It handles server crash via gRPC status errors but cannot distinguish a crashed server from a hung one without a configured deadline. Peer authentication requires a PKI (no SO_PEERCRED equivalent). It remains the strongest option for inter-VM, because gRPC provides network channel support, but the current adapter hardcodes Unix-domain endpoints and requires endpoint/configuration changes before that path is available. It is appropriate for QM-to-QM communication where safety certification is not required.

Option D (LoLa Message Passing Abstraction) meets the R1 library-capability assessment: its design supports fixed resource bounds, preallocation, and pool/monotonic allocation, and the communication module documents safety-oriented quality tooling and ASIL-B qualification. Its indicative POC latency of approximately 160 microseconds is the second-lowest measured value, but still represents significant IPC overhead and should remain a warning rather than an unqualified pass. The application code is OS-agnostic; the framework provides the OS-specific transport backend (Unix domain socket on Linux, QNX message passing on QNX). The low-level message-passing prototype provides feasibility evidence for the intended skeleton: non-blocking SendWithCallback / Reply decoupling, application-level request_id multiplexing for concurrent threads, application-level bounded waiting, and detection of server death via socket EOF. Typed error propagation, mandatory timeout APIs, and backend-specific bounded-notification behavior remain implementation work. The selected library does not support inter-VM communication: its current backends are local, and a new framework backend would be required. The IConnection abstraction may provide a migration direction, but this has not been demonstrated and is not evidence of current library support.

6. Decision#

Option D — LoLa Message Passing Abstraction is selected as the IPC transport for the score-crypto daemon, using score::message_passing as its current implementation basis.

7. Consequences#

Accepted trade-offs:

Indicative POC measurements show ~160 µs for message_passing, ~130 µs for LoLa Full SOA (async), and ~430 µs for gRPC. These figures are not a controlled benchmark and must be validated with the proper implementation using a common platform, payload, concurrency, warm-up, and measurement method. The observed gap between LoLa Full SOA and message_passing is partly attributed to configuration: the LoLa Full SOA asynchronous prototype was measured with QM-only settings, whereas the message_passing prototype was configured for mixed usage (truly_async=true). Configuring message_passing for QM-only reduced its observed round-trip to ~140 µs; the remaining difference was not analyzed. Throughput would be higher with LoLa Full SOA due to shared memory, but this is not a control-plane requirement: larger data transfers are expected to use a dedicated data plane.
The selected library does not currently support inter-VM communication. The existing IConnection abstraction may guide a second, dedicated IPC mechanism, but that migration path has not been demonstrated and remains separate follow-up work.

Production implementation follow-up:

The formal implementation of IConnection and IControlServer interfaces replacing the gRPC adapter is still pending will be taken up.