Benchmarks

Flow-Like includes Criterion benchmarks for single-execution latency, concurrent throughput, and allocator behavior. Use them to compare commits or deployment choices on the same machine.

Benchmark numbers are not a service guarantee

The reference results below are a historical internal snapshot. They record the broad machine class, but not an exact CPU model, commit SHA, compiler version, power mode, or thermal state. Treat them as orientation and rerun the checked-in harness for decisions about current performance.

What the harness measures

The throughput benchmark:

• loads the checked-in test app and board from tests/flow;
• initializes the current node catalog and local object stores;
• suppresses normal execution logs;
• warms the runtime before sampling;
• runs the same board concurrently at configured in-flight levels;
• reports Criterion latency and element-throughput distributions.

It measures in-process workflow execution. It does not include ingress, external APIs, remote storage, model inference, databases, queueing, network latency, container startup, or an end-user request path.

Historical reference snapshot

The previous internal run used:

| Setting | Recorded value | | --- | --- | | CPU | 16-core Apple M-series machine | | Memory | 32 GB | | Operating system | macOS | | Build | Optimized benchmark profile with thin LTO | | Allocator | mimalloc | | Workload | Small checked-in workflow; two nodes and five pins |

Four worker threads

| Concurrent executions | Approximate throughput | Approximate batch latency | | ---: | ---: | ---: | | 128 | 65,000 exec/s | 2.0 ms | | 512 | 100,000 exec/s | 5.1 ms | | 1,024 | 112,000 exec/s | 9.1 ms | | 2,048 | 121,000 exec/s | 17 ms | | 4,096 | 123,000 exec/s | 33 ms | | 8,192 | 124,000 exec/s | 66 ms |

Sixteen worker threads

| Concurrent executions | Approximate throughput | Approximate batch latency | | ---: | ---: | ---: | | 128 | 60,000 exec/s | 2.1 ms | | 512 | 140,000 exec/s | 3.6 ms | | 1,024 | 177,000 exec/s | 5.7 ms | | 4,096 | 228,000 exec/s | 18 ms | | 8,192 | 238,000 exec/s | 35 ms | | 32,768 | 241,000 exec/s | 135 ms | | 65,536 | 244,000 exec/s | 269 ms |

These values describe this synthetic in-process workload. A workflow that performs HTTP requests, model inference, storage, database work, or substantial serialization will be dominated by those operations.

Allocator snapshot

The historical high-concurrency allocator run recorded approximately:

| Allocator | Throughput at 1,024 concurrent executions | | --- | ---: | | mimalloc | 222,000 exec/s | | System allocator | 179,000 exec/s |

Allocator effects vary by platform, allocation pattern, and concurrency. Run both variants on the deployment target rather than assuming the same percentage improvement.

Run the benchmarks

Run commands from the repository root.

Throughput sweep

FL_WORKER_THREADS=4 \
FL_CONCURRENCY_LIST="128,512,1024,2048,4096,8192" \
FL_MEASURE_SECS=10 \
RUST_LOG=off \
cargo bench -p flow-like-catalog \
  --bench throughput_bench \
  --features mimalloc \
  -- peak_throughput

Raw throughput

FL_WORKER_THREADS=4 \
FL_MAX_IN_FLIGHT=32 \
RUST_LOG=off \
cargo bench -p flow-like-catalog \
  --bench throughput_bench \
  --features mimalloc \
  -- raw_throughput

Allocator comparison

RUST_LOG=off cargo bench -p flow-like-catalog \
  --bench allocator_bench \
  -- --save-baseline system

RUST_LOG=off cargo bench -p flow-like-catalog \
  --bench allocator_bench \
  --features mimalloc \
  -- --baseline system

The helper script runs both variants and formats their output:

bash packages/catalog/benches/compare_allocators.sh

All catalog benchmarks

RUST_LOG=off cargo bench -p flow-like-catalog --features mimalloc

The first run may spend significant time compiling the optimized benchmark profile. Later runs reuse Cargo artifacts unless relevant code or features changed.

Configuration

| Environment variable | Default | Purpose | | --- | --- | --- | | FL_BOARD_ID | Checked-in benchmark board | Board loaded from the test app | | FL_START_ID | Checked-in start node | Entry node executed by the benchmark | | FL_APP_ID | Checked-in test app | App directory under the test store | | FL_TESTS_DIR | ../../tests from the package | Local benchmark object store | | FL_WORKER_THREADS | Logical CPU count | Tokio runtime worker threads | | FL_MAX_BLOCKING_THREADS | Worker threads × 4 | Tokio blocking-thread ceiling | | FL_CONCURRENCY_LIST | Automatic sweep | Comma-separated in-flight levels | | FL_MAX_CONCURRENCY | Logical CPU count × 8 | Maximum automatic sweep value | | FL_MEASURE_SECS | 10 | Criterion measurement duration per level | | FL_MAX_IN_FLIGHT | Logical CPU count × 4 | In-flight tasks in the raw benchmark | | RUST_LOG | Inherited environment | Set to off for low-noise measurements |

Make a comparison reproducible

Record these details with every result:

1. Commit SHA and whether the worktree is clean.
2. Rust and Cargo versions.
3. Operating system, kernel, CPU model, core topology, and memory.
4. Power mode, virtualization, and container limits.
5. Allocator and Cargo feature set.
6. Board, start node, app ID, and any changes to the test fixture.
7. Worker, blocking-thread, concurrency, and measurement settings.
8. Criterion estimate and confidence interval, not only the fastest sample.

Compare two commits on the same machine with the same fixture and environment. Alternating baseline and candidate runs helps reveal thermal or background-load drift.

Interpret results

• Single execution isolates the overhead of one small in-process run.
• Throughput shows how efficiently the runtime uses concurrency.
• Batch latency grows with queue depth even while throughput improves.
• Peak throughput is not the same as a safe production operating point.
• External-node workflows require end-to-end benchmarks that include their actual dependencies.

Do not compare these numbers directly with another product unless the workload, durability, logging, retries, isolation, hardware, and measurement boundary are equivalent.

Contributing a benchmark

1. Add or update a target under packages/catalog/benches/.
2. Keep the workload deterministic and check in its fixture.
3. Document the measurement boundary and environment variables.
4. Include before-and-after Criterion output for performance changes.
5. Explain any change in behavior, durability, or correctness that accompanies the performance result.