Storage Providers

Flow-Like stores application metadata, content, and execution logs in object storage. The API also issues credentials scoped to a user, application, or run. Those are related but separate configuration concerns:

STORAGE_PROVIDER selects the object-store protocol used by the API.
RUNTIME_CREDENTIALS_PROVIDER selects how scoped credentials are created.

If RUNTIME_CREDENTIALS_PROVIDER is not present, the API uses STORAGE_PROVIDER. Do not define either variable as an empty string: an explicit empty value is a configuration error.

For a complete deployment configuration, use the Docker Compose storage guide or the Kubernetes storage guide. This page focuses on local development and provider-specific tests.

Supported configurations

Backing store	`STORAGE_PROVIDER`	Runtime credentials	Notes
Amazon S3	`aws`	`aws`	Uses AWS STS and `RUNTIME_ROLE_ARN` for scoped credentials
Azure Blob Storage	`azure`	`azure`	Uses time-limited Azure SAS credentials
Google Cloud Storage	`gcp`	`gcp`	Uses signed, scoped GCS credentials
Cloudflare R2	`aws`	`r2`	S3-compatible storage plus R2’s temporary-credentials API
MinIO or generic S3	`aws`	Provider-dependent	Works as a backing store through a custom S3 endpoint; scoped credentials need a separately supported mechanism

The API Cargo features are aws, azure, gcp, and r2. There is no provider-specific minio feature.

Common bucket names

Every provider needs content and log storage. Metadata can share the content bucket when META_BUCKET is omitted.

CONTENT_BUCKET=flow-like-content
META_BUCKET=flow-like-meta
LOG_BUCKET=flow-like-logs

Provider-specific names such as AWS_CONTENT_BUCKET, AZURE_CONTENT_CONTAINER, and GCP_CONTENT_BUCKET override the generic names.

Amazon S3

STORAGE_PROVIDER=aws
RUNTIME_CREDENTIALS_PROVIDER=aws

AWS_REGION=eu-central-1
AWS_ACCESS_KEY_ID=replace-me
AWS_SECRET_ACCESS_KEY=replace-me
CONTENT_BUCKET=flow-like-content
META_BUCKET=flow-like-meta
LOG_BUCKET=flow-like-logs

RUNTIME_ROLE_ARN=arn:aws:iam::123456789012:role/FlowLikeRuntimeRole

The backing store can use static environment credentials, an instance role, or the normal AWS web-identity credential chain. RUNTIME_ROLE_ARN is additionally required when the API must assume a role to issue short-lived, prefix-scoped credentials.

For a non-AWS S3 endpoint, add:

AWS_ENDPOINT=http://localhost:9000
AWS_USE_PATH_STYLE=true

Flow-Like reads the standard AWS_* variables for S3-compatible storage. It does not read MINIO_* variables.

Azure Blob Storage

STORAGE_PROVIDER=azure
RUNTIME_CREDENTIALS_PROVIDER=azure

AZURE_STORAGE_ACCOUNT_NAME=flowlikedev
AZURE_STORAGE_ACCOUNT_KEY=replace-me
AZURE_CONTENT_CONTAINER=flow-like-content
AZURE_META_CONTAINER=flow-like-meta
AZURE_LOG_CONTAINER=flow-like-logs

The account key is used to build stores and sign scoped SAS credentials. Keep the account key on the API; clients and executors should receive only the scoped credentials generated for their work.

Google Cloud Storage

STORAGE_PROVIDER=gcp
RUNTIME_CREDENTIALS_PROVIDER=gcp

GCP_PROJECT_ID=my-project
GOOGLE_APPLICATION_CREDENTIALS_JSON={"type":"service_account","project_id":"my-project"}
GCP_CONTENT_BUCKET=flow-like-content
GCP_META_BUCKET=flow-like-meta
GCP_LOG_BUCKET=flow-like-logs

GOOGLE_APPLICATION_CREDENTIALS_JSON is the service-account JSON itself, not a path to a key file. The API uses it to create signed, scoped credentials.

Cloudflare R2

R2 uses the S3 protocol for ordinary object-store access, but its own temporary-credentials API for scoped runtime access:

STORAGE_PROVIDER=aws
RUNTIME_CREDENTIALS_PROVIDER=r2

AWS_ENDPOINT=https://ACCOUNT_ID.r2.cloudflarestorage.com
AWS_REGION=auto
AWS_USE_PATH_STYLE=true
AWS_ACCESS_KEY_ID=replace-me
AWS_SECRET_ACCESS_KEY=replace-me

R2_ENDPOINT=https://ACCOUNT_ID.r2.cloudflarestorage.com
R2_ACCOUNT_ID=replace-me
R2_ACCESS_KEY_ID=replace-me
R2_SECRET_ACCESS_KEY=replace-me
R2_API_TOKEN=replace-me

CONTENT_BUCKET=flow-like-content
META_BUCKET=flow-like-meta
LOG_BUCKET=flow-like-logs

MinIO and other S3-compatible stores

MinIO can be used as the backing object store:

STORAGE_PROVIDER=aws
AWS_ENDPOINT=http://localhost:9000
AWS_REGION=us-east-1
AWS_USE_PATH_STYLE=true
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=replace-me
CONTENT_BUCKET=flow-like-content
META_BUCKET=flow-like-meta
LOG_BUCKET=flow-like-logs

This configuration covers object access only. Do not assume that an S3-compatible server implements the AWS STS behavior used by Flow-Like’s aws runtime-credential provider. For any deployment that sends scoped credentials to clients or remote executors, verify the provider-specific credential path end to end.

Tests

Serialization, policy-shape, and other non-networked credential tests run without cloud resources:

cargo test -p flow-like --lib credentials
cargo test -p flow-like-api --features full --lib credentials

Provider integration tests are ignored by default because they use real credentials and storage:

# Run every ignored credential test with all providers compiled.
cargo test -p flow-like-api --features full --lib credentials -- --ignored

# Compile and run one provider's ignored tests.
cargo test -p flow-like-api --features aws --lib credentials -- --ignored
cargo test -p flow-like-api --features azure --lib credentials -- --ignored
cargo test -p flow-like-api --features gcp --lib credentials -- --ignored

R2 currently has non-networked credential tests but no ignored live-provider suite in packages/api/src/credentials/r2_credentials.rs.

Troubleshooting

Unknown runtime credentials provider — ensure RUNTIME_CREDENTIALS_PROVIDER is absent or one of the compiled providers; do not leave it explicitly empty.
S3 signature mismatch — verify AWS_REGION, endpoint scheme, and AWS_USE_PATH_STYLE.
Missing logs — configure LOG_BUCKET or the provider-specific log bucket/container variable.
Backing-store access works but scoped execution fails — check the runtime credential provider separately. Successful S3 reads do not prove that STS, R2 temporary credentials, Azure SAS, or GCP signing is configured.