Benchmarks

Flow-Like includes Criterion benchmarks for single-execution latency, concurrent throughput, and allocator behavior. Use them to compare commits or deployment choices on the same machine.

Benchmark numbers are not a service guarantee

The reference results below are a historical internal snapshot. They record the broad machine class, but not an exact CPU model, commit SHA, compiler version, power mode, or thermal state. Treat them as orientation and rerun the checked-in harness for decisions about current performance.

What the harness measures

The throughput benchmark:

• loads the checked-in test app and board from tests/flow;
• initializes the current node catalog and local object stores;
• suppresses normal execution logs;
• warms the runtime before sampling;
• runs the same board concurrently at configured in-flight levels;
• reports Criterion latency and element-throughput distributions.

It measures in-process workflow execution. It does not include ingress, external APIs, remote storage, model inference, databases, queueing, network latency, container startup, or an end-user request path.

Historical reference snapshot

The previous internal run used:

Setting	Recorded value
CPU	16-core Apple M-series machine
Memory	32 GB
Operating system	macOS
Build	Optimized benchmark profile with thin LTO
Allocator	mimalloc
Workload	Small checked-in workflow; two nodes and five pins

Four worker threads

Concurrent executions	Approximate throughput	Approximate batch latency
128	65,000 exec/s	2.0 ms
512	100,000 exec/s	5.1 ms
1,024	112,000 exec/s	9.1 ms
2,048	121,000 exec/s	17 ms
4,096	123,000 exec/s	33 ms
8,192	124,000 exec/s	66 ms

Sixteen worker threads

Concurrent executions	Approximate throughput	Approximate batch latency
128	60,000 exec/s	2.1 ms
512	140,000 exec/s	3.6 ms
1,024	177,000 exec/s	5.7 ms
4,096	228,000 exec/s	18 ms
8,192	238,000 exec/s	35 ms
32,768	241,000 exec/s	135 ms
65,536	244,000 exec/s	269 ms

These values describe this synthetic in-process workload. A workflow that performs HTTP requests, model inference, storage, database work, or substantial serialization will be dominated by those operations.

Allocator snapshot

The historical high-concurrency allocator run recorded approximately:

Allocator	Throughput at 1,024 concurrent executions
mimalloc	222,000 exec/s
System allocator	179,000 exec/s

Allocator effects vary by platform, allocation pattern, and concurrency. Run both variants on the deployment target rather than assuming the same percentage improvement.

Run the benchmarks

Run commands from the repository root.

Throughput sweep

FL_WORKER_THREADS=4 \
FL_CONCURRENCY_LIST="128,512,1024,2048,4096,8192" \
FL_MEASURE_SECS=10 \
RUST_LOG=off \
cargo bench -p flow-like-catalog \
  --bench throughput_bench \
  --features mimalloc \
  -- peak_throughput

Raw throughput

FL_WORKER_THREADS=4 \
FL_MAX_IN_FLIGHT=32 \
RUST_LOG=off \
cargo bench -p flow-like-catalog \
  --bench throughput_bench \
  --features mimalloc \
  -- raw_throughput

Allocator comparison

RUST_LOG=off cargo bench -p flow-like-catalog \
  --bench allocator_bench \
  -- --save-baseline system

RUST_LOG=off cargo bench -p flow-like-catalog \
  --bench allocator_bench \
  --features mimalloc \
  -- --baseline system

The helper script runs both variants and formats their output:

bash packages/catalog/benches/compare_allocators.sh

All catalog benchmarks

RUST_LOG=off cargo bench -p flow-like-catalog --features mimalloc

The first run may spend significant time compiling the optimized benchmark profile. Later runs reuse Cargo artifacts unless relevant code or features changed.

Configuration

Environment variable	Default	Purpose
FL_BOARD_ID	Checked-in benchmark board	Board loaded from the test app
FL_START_ID	Checked-in start node	Entry node executed by the benchmark
FL_APP_ID	Checked-in test app	App directory under the test store
FL_TESTS_DIR	../../tests from the package	Local benchmark object store
FL_WORKER_THREADS	Logical CPU count	Tokio runtime worker threads
FL_MAX_BLOCKING_THREADS	Worker threads × 4	Tokio blocking-thread ceiling
FL_CONCURRENCY_LIST	Automatic sweep	Comma-separated in-flight levels
FL_MAX_CONCURRENCY	Logical CPU count × 8	Maximum automatic sweep value
FL_MEASURE_SECS	10	Criterion measurement duration per level
FL_MAX_IN_FLIGHT	Logical CPU count × 4	In-flight tasks in the raw benchmark
RUST_LOG	Inherited environment	Set to off for low-noise measurements

Make a comparison reproducible

Record these details with every result:

1. Commit SHA and whether the worktree is clean.
2. Rust and Cargo versions.
3. Operating system, kernel, CPU model, core topology, and memory.
4. Power mode, virtualization, and container limits.
5. Allocator and Cargo feature set.
6. Board, start node, app ID, and any changes to the test fixture.
7. Worker, blocking-thread, concurrency, and measurement settings.
8. Criterion estimate and confidence interval, not only the fastest sample.

Compare two commits on the same machine with the same fixture and environment. Alternating baseline and candidate runs helps reveal thermal or background-load drift.

Interpret results

• Single execution isolates the overhead of one small in-process run.
• Throughput shows how efficiently the runtime uses concurrency.
• Batch latency grows with queue depth even while throughput improves.
• Peak throughput is not the same as a safe production operating point.
• External-node workflows require end-to-end benchmarks that include their actual dependencies.

Do not compare these numbers directly with another product unless the workload, durability, logging, retries, isolation, hardware, and measurement boundary are equivalent.

Contributing a benchmark

1. Add or update a target under packages/catalog/benches/.
2. Keep the workload deterministic and check in its fixture.
3. Document the measurement boundary and environment variables.
4. Include before-and-after Criterion output for performance changes.
5. Explain any change in behavior, durability, or correctness that accompanies the performance result.