Workflows
Durable workflows, written in Rust.
Some jobs outlive the process that started them. Think of a provisioning run that is halfway through when a deploy restarts it, or an order that has to wait three days for someone to approve it. A workflow on Etorm is an ordinary program compiled to WebAssembly, and everything it does goes into an append-only history. After a crash, a restart or a long sleep, it picks up from the step it was on.
Who runs what
You write workflows and activities in Rust and run them in workers. A worker is one of your own Compute processes. The engine keeps the histories and hands out tasks, and it never runs your code. Everything the engine keeps is an object in the namespace's Storage bucket. The architecture page has the wider picture.
- Your worker is a Compute process in your project. It polls task queues, replays workflow code and runs activities.
- The namespace is the unit of isolation. It has its own bucket, processes, endpoint, limits and retention.
- Task queues carry two kinds of task. A workflow task asks a worker to run the workflow code on the new events. An activity task asks it to run one activity.
- The history is everything a workflow did, as events numbered from 1 that never change. The code runs again against it to find out where it was.
- The bucket holds histories, queues, timers and schedules as objects.
Four resources describe a namespace and what it knows about.
| Resource | What it is |
|---|---|
| WorkflowNamespace | A namespace with its endpoint, limits and retention. |
| WorkflowType | A workflow the namespace knows by name. |
| WorkflowActivity | An activity the namespace knows by name. |
| WorkflowSchedule | Starts a workflow on a schedule. |
Why workflow code has to be deterministic
A workflow keeps no checkpoint of its own. Its history is its state. Every step it takes is an event, a batch of events is stored all at once or not at all, and an event counts only once it is stored. To find out where a workflow was, a worker runs its code again from the top against that history.
That only works if the code makes the same decisions every time it is replayed. So workflow code has no clock, randomness or I/O of its own. When it asks for the time, a random number or a sleep, the answer comes out of the history. Anything that touches the outside world, an HTTP call to another service for example, belongs in an activity.
If you deploy code that no longer matches a history, the task fails and the workflow waits. Deploy a fix and it resumes. The history itself is never rewritten.
Activities, timers and schedules
An activity is retried by its retry policy until it succeeds or reaches a limit, and only the final result goes into the history. Since we retry until the activity reports back, an activity runs at least once and sometimes more than once, for instance when a worker dies after doing the work but before reporting it. Write activities so that running one twice does no harm.
A timer fires at an absolute time and never early. If a namespace has been idle and several timers fell due in the meantime, they fire together when it wakes, oldest first.
A schedule starts workflows on a cron expression, an interval or calendar times. For a given schedule and time, the start happens once.
Starts, signals, cancels and terminates each carry a request id. If your client times out and sends the same request again, the effect is not repeated.
Namespaces and identity
Each namespace has its own bucket, processes and rate limits, and namespaces share no state. When a request would go over a limit it is refused with ResourceExhausted, and the error names the limit.
Every history event records who caused it, taken from the caller's token. Payloads are never logged.
We kept your code out of the engine on purpose, so a workflow can only do what its worker's identity is allowed to do. The other side of that is that you run the workers yourself.
Limits
| Property | Value |
|---|---|
| History | At most 50 000 events and 32 MiB per workflow. |
| Payload | At most 2 MiB; a request at most 4 MiB. |
| Lifetime | 365 days by default, 5 years at most. Timers up to 5 years. |
| Per workflow | 100 pending activities and 1 000 timers by default; 50 events per second. |
| Names | Workflow ids, type names and task queues up to 255 bytes. |
When something fails
The engine acknowledges nothing it has not stored. Most of the cases below come down to a task timing out and somebody retrying it.
| What fails | What happens |
|---|---|
| An engine process | It is restarted or restored. Polls and held tasks are lost, and workers retry them. No history is lost, because an event counts only once it is stored. |
| A worker in the middle of an activity | The activity times out and is retried elsewhere. Activities run at least once. |
| A worker in the middle of workflow code | The task times out and is retried. The history never saw what the lost worker was doing. |
| Determinism | If changed code no longer matches the history, the task fails and the workflow waits. A fixed deploy resumes it. |
| Storage | Writes are refused. Nothing is acknowledged that was not stored. |
| A namespace you delete | Its processes and bucket are deleted, and its data is gone at once. |
If you have a job that keeps falling over halfway through, tell us about it.