Job lifecycle
Worker architecture
Job claiming
Workers claim jobs atomically from the queue using a claim epoch. If a job is reclaimed (e.g. after a heartbeat timeout), the original worker detects the epoch mismatch and stops processing. This prevents duplicate execution in multi-worker deployments.Heartbeats
Workers send heartbeats while processing jobs. If a worker stops sending heartbeats (crash, network failure) after the specified timeout, the job becomes eligible for another worker to claim and resume from the last checkpoint.Concurrency
Workflow execution has multiple levels of concurrency:- Jobs per worker — each worker processes multiple jobs concurrently. Configured via the max concurrency setting in workflow settings.
- Parallel steps per job — independent steps within a job execute simultaneously. Configured via the max parallel steps setting in workflow settings.
- forEach parallelism — iterations within a forEach step can execute concurrently. Configured per step in the forEach configuration.
Guarantees
Data processing & storage
DuckDB
Workflow steps execute on DuckDB, a high-performance columnar database. DuckDB runs in-process on the worker and handles SQL transformations, bulk data transfer to and from the app database, and intermediate storage operations. No external database connection is required for data processing. Step resource limits scale with worker configuration — see configuration for details.Step results
Step outputs are stored as Parquet files in the app’s configured workflow storage (i.e. S3 on AWS) or on the local filesystem during development. Each job run’s data is logically isolated from other concurrent runs. Step results are available for querying and debugging, and are automatically cleaned up after the configured TTL (default 48 hours).Monitoring
Track workflow execution through the platform’s monitoring tools:- Job status: Current state, start time, duration, step progress
- Step details: Per-step execution time, row counts, errors
- Logs: Structured logs for each step execution
- History: Past runs with outcomes and timing