Storage Configuration

Flow-Like requires S3-compatible object storage for workflow data, execution state, and logs. This page covers storage configuration specific to Kubernetes deployments.

Storage Buckets

Flow-Like requires S3-compatible object storage for workflow data, execution state, and logs. This page covers storage configuration specific to Kubernetes deployments.

Required Buckets

All three buckets (meta, content, logs) must be configured. Missing the logs bucket will cause executions to fail when trying to store execution logs.

| Bucket | Purpose | Recommended Storage Class | |--------|---------|---------------------------| | Meta | App metadata, execution state | S3 Express One Zone | | Content | User files, workflow data | Standard S3 | | Logs | Execution logs | Standard S3 / S3 IA |

AWS S3 Configuration

Basic Setup

# In your Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
  name: flow-like-s3
  namespace: flow-like
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "AKIAIOSFODNN7EXAMPLE"
  AWS_SECRET_ACCESS_KEY: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  AWS_REGION: "us-west-2"
  META_BUCKET: "flow-like-meta"
  CONTENT_BUCKET: "flow-like-content"
  LOG_BUCKET: "flow-like-logs"

S3 Express One Zone

S3 Express One Zone provides significantly better performance for metadata-heavy workloads:

| Metric | Standard S3 | S3 Express | |--------|-------------|------------| | Latency | 100-200ms | 1-10ms | | Request Cost | $0.0004/1k | $0.0002/1k | | Storage Cost | $0.023/GB | $0.16/GB |

Best for: Execution state store, app metadata, high-frequency reads/writes.

Express buckets are automatically created in a specific availability zone:

# Create Express bucket in us-west-2 AZ 1
aws s3api create-bucket \
  --bucket flow-like-meta--usw2-az1--x-s3 \
  --create-bucket-configuration \
    LocationConstraint=us-west-2 \
    Location={Type=AvailabilityZone,Name=usw2-az1} \
    Bucket={DataRedundancy=SingleAvailabilityZone,Type=Directory}

Enable Express mode via environment variable:

apiVersion: v1
kind: ConfigMap
metadata:
  name: flow-like-storage-config
  namespace: flow-like
data:
  META_BUCKET_EXPRESS_ZONE: "true"
  CONTENT_BUCKET_EXPRESS_ZONE: "false"
  LOGS_BUCKET_EXPRESS_ZONE: "false"

Availability Zone

S3 Express buckets are single-AZ. Deploy your API pods in the same AZ for optimal latency. Use pod affinity or topology spread constraints.

IAM Permissions

For S3 Express buckets, you need additional permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::flow-like-*",
        "arn:aws:s3:::flow-like-*/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3express:CreateSession"
      ],
      "Resource": [
        "arn:aws:s3express:*:*:bucket/flow-like-*--*--x-s3"
      ]
    }
  ]
}

Execution State Store

The execution state store tracks running workflows and their events. By default, it uses the meta bucket for storage.

# ConfigMap
data:
  # Uses META_BUCKET by default, falls back to S3_STATE_BUCKET
  EXECUTION_STATE_BACKEND: "s3"

This reuses the meta bucket configuration, so you don't need a separate bucket. If using S3 Express for the meta bucket, the state store automatically benefits from the lower latency.

Cost Optimization

S3 Express has higher storage costs but lower request costs. For execution state (many small reads/writes), Express is typically cheaper overall.

Scoped Runtime Credentials

Flow-Like generates scoped credentials for every workflow execution using STS AssumeRole. This ensures users can only access their own prefix-isolated storage paths.

STS Compatibility Required

Scoped credentials require credential scoping support from your S3-compatible provider:

• AWS S3 ✅ Full support (STS AssumeRole)
• MinIO ✅ Supported (requires STS configuration)
• Cloudflare R2 ✅ Supported (uses proprietary temp credentials API)
• Hetzner Object Storage ❌ Not supported

Providers without credential scoping support cannot be used with Flow-Like.

Setup

1. Create a runtime role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::flow-like-content",
        "arn:aws:s3:::flow-like-content/*",
        "arn:aws:s3:::flow-like-meta",
        "arn:aws:s3:::flow-like-meta/*",
        "arn:aws:s3:::flow-like-logs",
        "arn:aws:s3:::flow-like-logs/*"
      ]
    }
  ]
}

2. Add trust policy for the API service account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:flow-like:flow-like-api"
        }
      }
    }
  ]
}

3. Configure the API:

apiVersion: v1
kind: Secret
metadata:
  name: flow-like-runtime
  namespace: flow-like
type: Opaque
stringData:
  RUNTIME_ROLE_ARN: "arn:aws:iam::123456789012:role/FlowLikeRuntimeRole"

Credential Scoping

When RUNTIME_ROLE_ARN is set, each execution receives temporary credentials (valid 1 hour) scoped to:

| Path Pattern | Access | Purpose | |--------------|--------|---------| | apps/{app_id}/* | Read/Write | App data | | users/{user_id}/apps/{app_id}/* | Read/Write | User's app data | | runs/{app_id}/* | Write | Execution logs | | tmp/user/{user_id}/apps/{app_id}/* | Read/Write | Temporary files | | tmp/global/apps/{app_id}/* | Read | Shared temporary files |

Cloudflare R2

R2 is S3-compatible and supports prefix-scoped temporary credentials through Cloudflare's proprietary API:

apiVersion: v1
kind: Secret
metadata:
  name: flow-like-r2
  namespace: flow-like
type: Opaque
stringData:
  # R2 credentials for S3 API access
  R2_ACCESS_KEY_ID: "your-r2-access-key-id"
  R2_SECRET_ACCESS_KEY: "your-r2-secret-access-key"
  R2_ENDPOINT: "https://<account-id>.r2.cloudflarestorage.com"
  # R2 Temp Credentials API (required for scoped credentials)
  R2_ACCOUNT_ID: "your-cloudflare-account-id"
  R2_API_TOKEN: "your-cloudflare-api-token"
  # Bucket names
  META_BUCKET: "flow-like-meta"
  CONTENT_BUCKET: "flow-like-content"
  LOG_BUCKET: "flow-like-logs"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: flow-like-storage-config
  namespace: flow-like
data:
  STORAGE_PROVIDER: "r2"

R2 API Token Setup

The API token needs the Workers R2 Storage:Edit permission:

1. Go to Cloudflare Dashboard → Manage Account → API Tokens
2. Create a custom token with:
   • Permissions: Account → Workers R2 Storage → Edit
   • Account Resources: Include your account

MinIO (Self-hosted S3)

For air-gapped environments or local development:

apiVersion: v1
kind: Secret
metadata:
  name: flow-like-s3
  namespace: flow-like
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "minioadmin"
  AWS_SECRET_ACCESS_KEY: "minioadmin"
  AWS_ENDPOINT: "http://minio.flow-like.svc:9000"
  AWS_REGION: "us-east-1"
  META_BUCKET: "flow-like-meta"
  CONTENT_BUCKET: "flow-like-content"
  LOG_BUCKET: "flow-like-logs"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: flow-like-storage-config
  namespace: flow-like
data:
  AWS_USE_PATH_STYLE: "true"
  # Express zone not supported with MinIO
  META_BUCKET_EXPRESS_ZONE: "false"

Environment Variables Reference

| Variable | Description | Default | |----------|-------------|---------| | STORAGE_PROVIDER | Storage backend (aws, r2, azure, gcp) | aws | | META_BUCKET | Bucket for app metadata and execution state | Required | | CONTENT_BUCKET | Bucket for user content and workflow data | Required | | LOG_BUCKET | Bucket for execution logs | Required | | META_BUCKET_EXPRESS_ZONE | Enable S3 Express for meta bucket | false | | CONTENT_BUCKET_EXPRESS_ZONE | Enable S3 Express for content bucket | false | | LOGS_BUCKET_EXPRESS_ZONE | Enable S3 Express for logs bucket | false | | RUNTIME_ROLE_ARN | IAM role ARN for scoped runtime credentials (AWS/MinIO) | Optional | | R2_ACCOUNT_ID | Cloudflare account ID for R2 temp credentials | R2 only | | R2_API_TOKEN | Cloudflare API token for R2 temp credentials | R2 only | | EXECUTION_STATE_BACKEND | State store backend (postgres, redis, s3) | postgres | | AWS_USE_PATH_STYLE | Use path-style URLs (for MinIO/R2) | false |