⚠ Switch to EXCALIDRAW VIEW in the MORE OPTIONS menu of this document. ⚠

Text Elements

job

id: primary key interval: int scheduled_at: date completed_at: date retry_policy: RetryPolicyEnum

Task Scheduler

grab jobs for time slot

on an interval

client

worker nodes

Load Balancer

Job Scheduler

Enqueue Service

jobs shard 2

jobs shard 0

jobs shard 1

Orchestrator

shard health, redirecting requests to shards

Task Scheduler

grab jobs for time slot

post job to queue

grab jobs

update job status?

in-memory message broker

output object store

distributed lock service

request lock on jobId

heartbeat

write new jobs

database links to this

read in data when processing job

input object store

write output to object store

database links to this for job output

in-memory because durability/retry logic is handled by the task scheduler along with the jobs database.

🗿