Executor

The Rust execution manager assigns each run to a preinitialized, single-use gVisor runner Pod. A separate gateway Pod enforces that run’s network permissions. The runner is destroyed after use and the manager prepares a replacement in the background.

The default configuration uses execution.isolationMode=per_run. Both interactive HTTP dispatch and the Redis queue bridge use this manager. The chart rejects the legacy Kubernetes Job backend for isolated execution.

Follow one execution

1. The manager creates a runner NetworkPolicy and its paired gateway policy before creating either Pod.
2. The gateway starts with no execution grant. The runner checks its gateway connection and prohibited cluster/node/metadata endpoints, initializes the trusted Rust catalog and JWT verification key, then waits for input.
3. Admission removes a ready slot and atomically claims the run in Redis with SET NX EX. An existing claim prevents another manager replica from assigning the same run.
4. The manager binds both Pods to the run and deadline, checks cancellation markers, configures the gateway once and sends a signed dispatch.
5. The runtime verifies the dispatch before fetching executable artifacts. It executes once and streams bounded events and terminal acknowledgement.
6. The manager confirms runner termination, removes the Pod pair and only then removes the restrictive policies.

The slot adapter shares the runner’s sandbox. Its HTTP server is a transport adapter; cancellation, deadlines and egress are enforced externally. No sandbox is reused across tenants or executions.

Capacity and warm reserve

Helm value	Default	Effect
executionManager.replicaCount	1	Independent manager partitions
executionManager.maxConcurrentExecutions	10	Active runs per manager
executionManager.warmPoolSize	2	Additional clean slots per manager
executionManager.warmPoolCreationConcurrency	2	Concurrent preparation per manager
executionManager.warmPoolMaxAgeSeconds	600	Unassigned slot lifetime
executionManager.workerThreads	2	Async supervisor workers
executionManager.queueBridge.replicaCount	1	Background dispatch consumers
executionManager.queueBridge.concurrency	10	Outstanding jobs per consumer

Set runner CPU, memory and temporary storage under executionManager.sandbox. The default runner requests 1 GiB of memory, has a one-CPU limit and a 256 MiB memory-backed temporary volume. Include the separate gateway and gVisor overhead in node sizing.

Each gateway requests 32 MiB and is limited to 128 MiB. Set kubelet podPidsLimit on execution nodes to bound process creation; the chart relies on that node setting for its portable PID limit. Pin executionManager.image.digest and executionManager.sandbox.image; scripts/resolve-images.py reads both from the published packages.

Configured concurrency is approximately manager replicas multiplied by the active limit. Sustainable throughput also depends on run duration and preparation rate: 100 executions per second with a 60-second average duration require about 6,000 concurrent executions. Adding replicas cannot compensate for exhausted nodes.

When no clean slot is available, immediate requests are refused without starting a run. Queue delivery can wait within its age budget. Monitor available warm slots and preparation failures as well as active execution counts.

An unavailable slot returns HTTP 429 with X-Execution-Admitted: false before streaming headers are sent. Queue bridges may retry that explicitly unstarted work within its original queue-age budget. Accepted work retains its deadline and admission slot if the requesting client disconnects.

Latency and long runs

Warm initialization removes process creation from the admission path. The Rust manager reuses asynchronous network connections and binds the two Pods in parallel. Kubernetes API requests, Redis claims, credential issuance and artifact preparation still contribute to start time. No few-millisecond start guarantee has been measured.

executor.timeout defaults to one hour. Separate startup, terminal and cleanup allowances cover supervision, and Pod deadlines remain in force if a manager disappears. Graceful manager shutdown stops new admission and allows accepted executions to drain.

The default RustFS session request is two hours. Actual credential expiry must cover queue wait, execution and the supervisor allowances. See Configuration before changing either the workflow or credential limits.

Cancellation and replay protection

Cancellation writes a shared ConfigMap marker before looking up assigned Pods. The manager shortens Pod deadlines and waits for kubelet-reported termination. It does not equate forced deletion of a Pod API object with a stopped sandbox. If a node or API partition prevents confirmation, cancellation reports failure and admission closes.

Redis claims survive manager replacement and expire after 24 hours plus the execution and supervisor allowances. A lost claim reply is not retried. Retain Redis data across upgrades; restoring a snapshot can remove claims for work that already ran. External side effects still require application-level idempotency.

Claims use exec:claims:v1:<namespace>:<release>:<sha256-run-id> and contain only a slot identifier. The manager checks Redis before becoming ready and uses two-second connection/command timeouts. TLS validates the certificate chain and hostname; connection-URL query overrides that could weaken these checks are rejected. See Security for a dedicated Redis ACL.

Budget Redis for retained claims as well as queues: 1,000 executions per second creates at least 86.4 million claim keys over 24 hours. Bundled Redis fsyncs AOF every second, so a crash can lose recent claims. Reconcile accepted work before resuming dispatch after Redis data loss or restoration.

The exec:jobs:v3 queue retains accepted delivery until trusted terminal confirmation. Ambiguous delivery and expired queue items are quarantined for reconciliation. Do not replay them solely because the client lost its connection.

Inspect execution

kubectl get pods -n flow-like -l app.kubernetes.io/component=execution-sandbox -o wide
kubectl get pods -n flow-like -l app.kubernetes.io/component=execution-egress -o wide
kubectl logs deployment/flow-like-execution-manager -n flow-like --tail=100
kubectl logs deployment/flow-like-queue-bridge -n flow-like --tail=100
kubectl port-forward service/flow-like-execution-manager 9000:9000 -n flow-like

Manager /ready reports supervisor availability. Inspect /metrics to establish whether warm capacity is actually available. Runtime dispatch and cancellation endpoints require the manager token and should remain private.

executor_warm_slots reports the ready reserve; metrics also show active jobs, admission capacity, slot preparation/retirement and assignment durations. Measure queue-to-first-node p50/p95/p99, completion throughput and slot replacement rate on the actual cluster, including cancellation and hour-long executions. Raising the reserve or preparation concurrency also increases Kubernetes API, CNI and node load.

For local protocol and controller tests, run from the repository root:

cargo test --locked -p flow-like-execution-manager

The suite uses fake APIs, Redis wire fixtures and harmless child processes for admission, cancellation, replay, transport and cleanup. It does not replace live gVisor, Cilium, storage or load qualification.

Trusted local workflows

execution.isolationMode=trusted_shared with execution.asyncBackend=http enables the existing reusable executor pool. Configure its replicas and bounded concurrency under executorPool. It shares a process between executions and does not meet the multi-tenant isolation requirement.

The source entry points are apps/backend/execution-manager/src/kubernetes/, apps/backend/kubernetes/executor/src/main.rs and packages/executor/. The Kubernetes runner uses /app/execution-slot to deliver one --once warm dispatch to the Rust executor.