How the clean-slate Elsa kernel fits together
A walk through the new execution kernel, from the three layers down to the rows a commit writes. The diagrams with step buttons can be played through; every step is also written out beside them. New to the words used here? Start with the companion The kernel in motion, which follows one order workflow through every part, with a visual glossary and a lab for the fence.
The kernel in one picture
The kernel is three layers and one storage contract. The actor runtime decides where things run. The durable layer makes every guarantee: nothing is lost, nothing applies twice, and a stale node can never write. The actor types on top do the actual work, and a workflow instance is one of them. Click a layer to see what it owns.
1 · Actor runtime
- Owns
- Placing actors on nodes, running each scope one pass at a time, wake-ups between nodes.
- Not
- Any guarantee about messages or state.
- Swap
- Built in: partitions. Integrators: Orleans, Proto.Actor, Akka.NET, Dapr actors.
2 · Durable actors
- Owns
- The durable inbox; a turn's new state and sent messages committed atomically; the fence against a stale owner; timers; background jobs; events; poison handling; the journal.
- Not
- Where an actor runs.
- Swap
- Never. Its storage can be swapped.
3 · Actor types
- Owns
- What an actor does with its messages. A workflow instance holds what all workflows share and runs an execution model for the rest.
- Not
- Persistence, placement.
- Swap
- Extended by adding actor types and execution models.
Persistence provider
- Owns
- Nine core guarantees: atomic commit, a fence inside it, conditional take, create-only inserts, an order key, reads in key order, reads by key, its own clock, failures told apart. The prototype's store also carries the node heartbeat, start subscriptions, held messages, delete and the projector's rows, each tested by the conformance suite.
- Not
- Any SQL in the kernel.
- Swap
- Any store that passes the conformance suite.
The idea in five sentences
- The durable layer knows nothing about workflows. It runs durable actors that own a piece of state and react to messages.
- An execution model turns an instance's state and incoming messages into new state and outgoing messages. It is the only extension point.
- Everything that crosses an instance boundary is a message: a start, a child finishing, a timer, an external event, a cancel, an alteration. No code reads another instance's state.
- Under the built-in runtime, actors are grouped into partitions. One node owns a partition at a time and is its only writer, so the hot path needs no reads, no per-instance lease and no read before a turn.
- A commit writes everything a turn changed in one transaction, and the owner batches the commits of many instances into one.
Words used here
- Actor
- Something with its own state that handles one message batch at a time. A workflow instance is one kind.
- Turn
- An instance's turn to run: it takes the messages in its inbox, runs as many activities as its persistence mode allows, changes its state and sends messages. Temporal calls this a workflow task. One turn is not one activity.
- Pass
- The runtime's call into a partition it hosts: it runs the turns that are due and commits them in batches. One pass holds many turns.
- Partition
- A fixed bucket of instances whose owner is their only writer. The original prototype uses 64 on server databases and 1 on SQLite; existing illustrations use that shape. The adopted 15 November experiment compares 1,024 and 4,096 fixed virtual partitions, provisionally 4,096; both fence variants preserve a gapless journal per virtual partition. Final fence granularity awaits the owner's decision after measurements; this experiment is not implemented.
- Epoch
- A counter on the partition row, raised every time a node takes the partition. Work started under an older epoch is thrown away.
- Position
- How many journal rows a partition has written (a turn at the Minimal history level writes none). The commit checks it, so two commits can never land at the same position.
- Fence
- The conditional update inside every commit: it lands only while the epoch and position are still the ones the owner expects.
- Lease
- How long the owner may hold the partition without writing (10 s). It decides who should write; the fence decides who can.
- Message
- A row in the inbox table: target, kind, payload, an order key
seq, optionally a due time (a timer) and a start spec (create the target if missing). - Job
- A background job: something slow an instance starts during a turn, whose result comes back later as a message while the instance keeps taking other messages.
- Journal
- One compact row per instance commit, written in the commit: the record of what happened. The viewer's tables are built from it.
- Persistence mode
- Where a workflow ends its turns: Ephemeral (only at a wait, if ever), Balanced (at each wait), Durable (after each activity). Chosen per workflow.
The seam between the runtime and the durable layer
Layer 2 hands the runtime a set of ownership scopes, one per partition, and the runtime decides which node hosts each one. The runtime runs a scope one pass at a time; a pass runs the turns that are due. Layer 2 does not trust it to be right: the fence catches a second activation on another node.
public interface IActorRuntime
{
ScopeId ScopeOf(ActorId actor); // the actor's partition
ValueTask StartAsync(IScopeHost host, CancellationToken ct);
ValueTask WakeAsync(ActorId actor, CancellationToken ct); // a hint: there is work
ValueTask StopAsync(CancellationToken ct);
}
public interface IScopeHost // implemented by layer 2
{
ValueTask<IScopeActivation?> ActivateAsync(ScopeId scope, CancellationToken ct); // takes the fence, or declines
}
public interface IScopeActivation : IAsyncDisposable
{
ValueTask WokenAsync(ActorId actor, CancellationToken ct);
ValueTask<TimeSpan?> RunAsync(CancellationToken ct); // one pass: turns until idle; when to call again
}
The prototype was split this way in round 4 stage A.
What is stored
Eight record kinds in the shared-store shape, shown as tables; EF and MongoDB implement equivalent records. The checkpoint holds owner state; the journal records committed history. Their current transaction inserts the recipient inbox with sender state, so this shape needs no separate outbox. The accepted provider-neutral rule records sends atomically with sender state and permits deduplicated later delivery through an outbox when cross-partition transactions are unavailable. Click a table.
These cards use the sketch's proposed record names for the existing shared-store shape. Non-owner queries, including lists and counts, use read models in the proposed design. The round 5 H paper separates that proposal from built behavior. The Storage model reference maps the physical EF tables and MongoDB collections, with lifecycles, writers, indexes and retention costs.
partition
The fence. epoch changes only on a take; position counts journal rows. Times are in milliseconds; under the default fence lease_until is written from the owner's clock and compared with database time at a take.
node
A heartbeat, so nodes can count each other and take a fair share of partitions. Liveness only, never correctness.
instance
state is opaque, owned by the model. jobs lists background jobs in flight. This checkpoint is owner state; the proposed non-owner list/count read path uses read models, not this row directly.
message
The inbox. seq from the database gives the order; the unique id makes adding idempotent; a future due makes it a timer or a retry; start creates the target if missing.
subscription
Routes external events to waiting instances (kind wait) or to a definition that starts one (kind start).
event
A published event. By default it reaches only waits committed before it; a wait that asks for kept events also gets events published within their window before it.
journal
One row per instance commit, written in that commit, according to the history level (Full by default). At Minimal no row is written.
projection
How far the projector has got. Its own row, so the projector never touches the fence row and never slows a commit.
Life of a turn
Follow one message from the moment someone adds it until its effects are durable, handed on and visible in the viewer. Use the buttons or click a step below the diagram.
- Add a message.
Anyone may insert one: the API, an event router, another partition's commit. The unique id makes it idempotent. A message from outside carries
keep_until, so a retried command gets a duplicate-key answer instead of a second effect. - Wake the owner.
The adder signals the owner, in process or through a notifier between nodes. It is only a hint; the poll is the guarantee.
- Poll.
A plain read: rows of the partition not seen at this epoch, not done, due within the horizon, in
seqorder. No locks and no watermark, so a row whose insert commits late is found next time. - Mark it seen.
A message that came by poll is not acted on until a fenced commit marks it
seen = 8. If this node dies, the next owner knows exactly which messages it was holding. Messages sent inside a partition are written as seen and skip this. - Run the turn.
The execution model gets the state from memory and the inbox in order. It changes state, sends, subscribes, starts jobs, records journal entries and emits live output. Nothing is read from the database.
- Commit.
The committer takes every finished turn of the partition: fence, instance rows, consumed messages deleted, sent messages inserted, journal rows. One transaction, 2 to 3 round trips on PostgreSQL, measured at 100 instances; rows per flush are capped (500 by default in the prototype); SQLite's calls grow with the rows.
- Hand over and release.
Only after the commit lands: a message to an instance in the same partition goes over in memory, and a held HTTP response is released to the caller. A caller never sees a response for work that was then lost.
- Project.
Later, the projector reads journal rows in position order per partition and writes the viewer's tables. The hot path never waits for it. The run view stays current by reading the journal tail past the projector (decided, not built); lists over projected rows lag it slightly.
The commit
The owner of partition p at epoch e, which believes the position is n, commits a batch of instance turns in one transaction, k of which write a journal row. Switch between the two PostgreSQL paths. The list groups possible writes by table. Their order can vary with the operations in the batch.
- Fence.
UPDATE partition SET position = n + k, lease_until = … WHERE id = p AND epoch = e AND position = n. Also renews the lease, so a busy partition needs no heartbeat. - Exactly one row changed? If not, roll back and reset. This check must come before COMMIT is sent.
- Insert or update the instance rows the batch changed, each update
WHERE revision = r. - Delete the consumed messages, or mark them
donewhen they carrykeep_until. Check the count. - Mark
seen = eon messages held in memory and not consumed. - Insert the sent messages.
Sendderives IDs from the sender, revision and ordinal; repeats are refused. Its fixed-width ordinal orders sends within one revision. The provider must preserve send order across revisions too. - Insert and delete subscriptions.
- Insert k journal rows at positions n + 1 … n + k.
- COMMIT.
When a commit fails
| What happened | What the owner does |
|---|---|
| Unknown outcome: the connection broke mid-commit | Reset: drop everything in memory for the partition and take it again. Recovery is the start-up path. |
| The fence matched nothing | Reset. Another node owns the partition now, or this node's memory is stale. |
Deadlock victim, SQLITE_BUSY (transient) | Retry the batch as it is. |
| Definite error: a value too long, a key violation | No reset. Commit the batch's instances one by one and park the one that fails (faulted, with the error). The rest go on. |
Ownership and takeover
Leases decide who should write. The fence decides who can. Play through a takeover where the old owner was only paused, the case that breaks most designs.
- A takes the partition.
The take succeeds only if the partition is free, its lease has expired, or A already owns it. Every take raises the epoch: now 7.
- A commits.
The fence checks epoch 7 and position 40 and moves the position to 43. The same update renews the lease. An idle owner sends an empty fenced commit as its heartbeat.
- A stops.
A long GC pause or a network split. A still believes it owns the partition and still holds instances in memory.
- The lease runs out.
Nobody has written, so nothing is wrong yet. A wrong lease costs availability, never correctness.
- B takes the partition.
Epoch 8, position 43. In the same transaction every message A had on trial gets one more attempt: that is how a message that kills its owner gets counted.
- B polls everything.
Every row with
seen < 8comes back, so B finds all A was holding. B builds memory from rows alone, and hands nothing over in memory until a poll comes back short. - A wakes and tries to commit.
Its commit is fenced on epoch 7. It matches no row, so nothing lands. A resets and drops its memory. This is the Orleans two-activations case, caught.
- B carries on.
B commits position 43 to 44. No two commits ever landed at the same position, and nothing prepared under epoch 7 got in.
- Fair share. A node counts the others from the
nodeheartbeats, holds its share of partitions and releases the surplus. - Reset is start-up. After any doubt the owner drops its memory and takes again. A successful take obtains the true position: relational stores wait behind the transaction; MongoDB may return a transient conflict and require a retry. A take that changes no row returns nothing.
- Failover becomes eligible after one lease (10 s by default); contention and recovery reads can add time. The epoch fence prevents a superseded owner from committing.
Messages, timers and poison
The inbox is the outbox: a message to another instance is a row inserted in the sender's own transaction, and nothing is relayed afterwards.
Order
A target handles its messages in seq order. A sender's commits are serial, so messages from one sender to one target arrive in the order sent.
Once in state
A state change and the delete of the message that caused it are one transaction. A message changes state exactly once, though a turn's code may run more than once after a crash.
Outside commands
Carry keep_until. The consuming commit marks them done and a sweep deletes them later, so a client retrying a handled command gets a duplicate-key answer, not a second effect. Messages between instances need no window: a repeat cannot exist.
Timers
A message with a future due. The poll returns database time; the owner anchors it and schedules on a monotonic clock. Cancel a timer by deleting its message, whose id the sender knows.
Creating instances
A message may carry a start spec: if its target does not exist, the owner creates it on delivery. Starting a child, starting on an event and signal-with-start are all this one mechanism.
Placement
A start without a caller's key goes into a partition the starting node owns and runs there, with no hop. A keyed start hashes the key to a partition. A child goes into its parent's partition, so a call tree commits together.
Poison
A new owner puts its predecessor's held messages on trial, one at a time by default (an optional larger group halves with every attempt). If that owner is lost too, the next take adds one attempt to exactly the messages on trial, so a poison message ends up alone and a death stops counting against its neighbours. Past the threshold it is parked and its instance faulted, so a message that kills its process cannot walk the cluster.
Edge cases
A message to an instance that does not exist and carries no start spec is dropped and journalled. A message to a finished instance is consumed and dropped. A turn that throws parks its instance at once; retries are the model's business.
Execution models
A workflow instance runs an execution model. The model is a plug-in with one job: given the state and the inbox, produce new state and outgoing messages. Everything else is the kernel's.
public interface IExecutionModel
{
string Key { get; } // "elsa.native", "bpmn", ...
int StateVersion { get; }
object CreateState(StartSpec start);
object UpgradeState(object state, int from);
ValueTask TurnAsync(ITurn turn, CancellationToken ct);
ValueTask<object?> ExecuteJobAsync(IJob job, CancellationToken ct);
InstanceSummary Describe(object state);
}
The proposed contract. The prototype already uses turn, job and pass; where its signatures still differ, its DESIGN.md lists most of them.
public interface ITurn
{
InstanceInfo Instance { get; }
IReadOnlyList<Message> Inbox { get; }
IAmbient Ambient { get; } // this activation only
object State { get; set; }
string Send(...); void Recall(id); InstanceRef Start(spec, placement, detached);
void StartJob(key, args); void CancelJob(key);
void Hold(message);
void Subscribe(key, wait, keep); void Unsubscribe(...);
string Publish(key, payload, keep); bool Redelivered(message);
void Record(entry); // journal
void Emit(channel, item); // live, not durable
void Rebind(definition); void SetStatus(...);
}
A persistence mode is just where a turn ends
A turn commits at its end and only there. A model that wants a durable boundary ends the turn and sends itself a "continue" message, which the owner hands over in memory. That is all a persistence mode is. Here are ten activities with no wait, measured on PostgreSQL:
How the known models sit on the contract
Only Native is built, as a spike; the others are the planned fit.
Native (Elsa)
State is the live activity frames, variables and scheduled work. Slow activities are jobs. Each container brings its own semantics (below). Built as a spike; it passes its own simulation checks.
BPMN
Hosted on Bpmn.Semantics, a pure interpreter: definition, token state and one event in; next state and commands out. A turn feeds it the message; its commands become jobs, timers, subscriptions and sends. State is tokens and scopes. A task is a job, so an interrupting boundary event can cancel it. A call activity is a start. A message that arrives early waits in event.
ETL
State is stage cursors and references to data held elsewhere. Partitions of work are children spread over partitions. Backpressure is a credit protocol between instances.
Agents
State is the conversation checkpoint (a Microsoft Agent Framework thread). A tool call is a job; streamed output is emitted; another agent is another instance; a human approval is a subscription.
Standalone jobs
Elsa.Jobs rebuilt on the kernel: a job is an actor type next to the workflow instance, so a .NET app gets durable jobs without workflows. A workflow's job is the same thing, in the same commit, with no bridge between stores.
Durable entities
State is the entity's value; operations are messages with a reply-to. The reviewers' test of a fifth model: it fits with no change.
Replay-style authoring
Temporal and Durable Task style code, or Restate's journaled awaits, fit as models on the same contract. The kernel imposes neither.
Containers and their semantics
The native model is one activity tree. Each container brings its own semantics, runtime state and kind of diagram, and containers nest freely. The kernel sees none of this; it sees turns. Decided 12 November.
| Container | What drives it | Runtime state | Semantics |
|---|---|---|---|
| Sequence, If, ForEach, While, Parallel | The previous child completing | An index; a fixed join | Built in. The structure says whether more can arrive, so no tokens |
| Flowchart | Outcomes choose the flows | Tokens (execution paths) and join arrivals per loop round | Graph engine: today's Elsa 4 Flowchart engine (dead-path joins, BPMN's gateway kinds) lifted into a library for the native model |
| StateMachine | Events fire guarded transitions; exit, transition and entry actions run | The active configuration of every region | Statechart engine: one event per turn; later events wait with Hold while an action is suspended. A native composite (recommended; #3815's design milestone decides), so it works as a root and nested |
The plan: extract the graph engine and port the prototype's Flowchart to it (by the prototype's own reasoning, its All join after a decision does not fire; the first step shows it with a test), a BPMN spike that hosts the Bpmn.Semantics interpreter on the kernel, then a StateMachine spike with nested states and parallel regions. Each step is gated by new simulation invariants and mutation checks, and none may move the cost table.
Background jobs
Models compose through messages only. A parent of any model starts a child of any model and gets a message when it finishes. A cancel goes to the children the parent itself started, so admitting a child and cancelling the parent cannot race. The kernel owns suspend and resume, terminating a tree, tenant checks on sends and trace headers.
Events and correlation
An external event (an approval, a payment notice, a BPMN message) must reach every instance waiting for its key, even when the event and the wait are committed at nearly the same moment. Each side checks for the other after its own commit.
The alternative: correlation as an ordinary instance per key
Built as a 50-line model with no kernel change: subscribing and publishing are messages to the key's instance, which holds the waits and the kept events. It passed every run. It costs a hop, measured on PostgreSQL for one waiter, one event:
| Version | Commits | Round trips |
|---|---|---|
| Tables (subscription, event, owed read) | 4 | 17 (13) |
| Key instance in the waiter's partition | 6 | 21 (15) |
| Key instance in another partition | 9 | 33 (24) |
In brackets: with the fence in the batch. Decided 12 November: the tables stay the kernel's one mechanism; the instance per key remains a sample model.
Synchronous HTTP
A workflow handles a request and writes the response itself. The response travels on the live Emit channel, not as a durable message: if it cannot reach the caller, the caller gets an error and the workflow is unaffected. Pick a case.
- The request arrives.
Any node can take it. The route table is built when definitions are published, so finding the workflow reads nothing.
- It runs inline.
The node places the new instance in a partition it owns and runs it in the request, with the request as
Ambient. No hop. - Respond and finish.
The workflow writes the response and ends without waiting. In Ephemeral mode nothing is committed: zero database calls.
- The request arrives.
Same as before: the route table is in memory.
- It runs inline.
The instance is placed locally and runs in the request.
- The turn commits.
The response is held until the commit of the turn that wrote it lands: 2 to 3 round trips of latency on PostgreSQL. A caller never sees a response for work that was then lost.
- The response is released.
Writing the response does not end the workflow; the request returns and the workflow goes on. A route can opt out and respond before the commit.
- A resume arrives at node A.
The workflow it resumes is waiting, and node B owns its partition.
- A adds a resume message.
It carries a reply address, node A and the request's id, and A keeps the request open.
- B runs the turn.
B finds the message by poll or wake hint, marks it seen and runs the turn.
- B commits.
The turn that wrote the response commits, so the response may now be released.
- B replies to A.
Over the runtime's transport between nodes, the same one wake-ups use. It is not durable. On one node this is all in process.
- A answers the request.
One hop paid, because B owns the instance. Moving the instance to the receiving node is left out unless measurements show the hop matters.
- A resume arrives at node A.
Node B owns the waiting workflow.
- A adds the resume message
and holds the request open with a reply address.
- Node B dies.
After its lease runs out another node takes the partition and resumes the workflow from the rows, as usual.
- The request ends with 503.
By default, never a hang and never two responses. The workflow's later output is not sent to a caller that is gone. In the prototype an owner lost on another node is noticed only at the deadline (504): that signal is not built yet.
- Waiting before responding (a background job, a short timer): the instance stays in memory on the receiving node and the request waits, up to the route's timeout.
- Client disconnect and timeout end the observation. They do not cancel the workflow, unless the route maps them to a cancel message.
- Streaming (chunked responses, server-sent events, an agent's tokens) uses the same channel. Large bodies stay out of state; the workflow holds a reference.
Deployment profiles
A deployment profile combines an actor runtime and a persistence provider; persistence mode remains the workflow author's choice. Release 1 adopts Embedded and Cluster with the same Native structured containers, Flowchart and StateMachine. Orleans requires proven demand. Log is later, planned, not designed; equivalent behavior and synchronous HTTP remain requirements for that future design. Pick a profile.
Persistence mode
Chosen per workflow by its author. Gives the workflow its guarantees.
EphemeralBalancedDurableActor runtime
Layer 1. Where actors run.
Built-in, one node, no transport Built-in, partitions shared over nodes Orleans adapter Planned: replicated-log leadershipPersistence provider
Behind IKernelStore. What persists the data.
SQLite (1 partition) PostgreSQL SQL Server (RCSI on) MySQL MongoDB 8 (replica set) Planned: replicated log (Zeebe-like)| Kernel concept | With a replicated log |
|---|---|
| Commit | a fenced batch appended to the partition's Raft-replicated log, acknowledged after consensus commitment |
| Fence and epoch | Raft term-based authority, enforced by the commit protocol |
| Lease and take | leader election |
| Journal | the log itself |
| Inbox and polling | log entries plus local state; no inbox-table polling |
| Projector | an exporter that reads committed log records past its position |
IKernelStore and ownership in IActorRuntime. The round 5 H paper proposes four contracts before code. The accepted send boundary records sends with sender state and permits deduplicated later delivery; logical per-virtual-partition journal positions remain distinct from physical log indexes.What it costs
EF Core commands per workflow run on SQLite, for three systems in one unit: Elsa 3 (3.10.1), Elsa 4 today (main) and the new kernel's native model. Hover a row for the three figures. On PostgreSQL the kernel's Balanced commit is 2 round trips with the fence in the batch, 3 with the checked fence (NATIVE.md section 5).
| Engine | Balanced run on the owner | Notes |
|---|---|---|
| PostgreSQL | 3 (2) | Round trips; in brackets the fence in the batch. |
| SQL Server | 4 (3) | Needs read-committed snapshot isolation. |
| MySQL | 7 (6) | Oracle's EF provider; upserts and INSERT IGNORE. |
| MongoDB 8 | 3 (2) | Needs a replica set for transactions; the store assigns the order key. |
| SQLite | 5 | Calls into the engine, where EF sends one command per row. This counts BEGIN and COMMIT too, so it is 5 here against the chart's 3 commands. |
This table counts a different unit from the chart above: round trips for the server databases, and engine calls (BEGIN and COMMIT included) for SQLite. Measured by the prototype's own cost runs, as reported in its findings; not re-measured for this record.
How it is tested
The model is attacked by a deterministic simulation: several nodes against one store, one random action at a time (takes, polls, turns, commits, crashes, a paused owner that wakes with stale memory, commits whose answer is lost, retried commands, clocks that disagree). After every action it checks these invariants. A failing seed replays exactly.
1 · One writer
No two commits land at the same position; nothing prepared under an old epoch lands.
2 · Atomic turn
State, consumed and sent messages, subscriptions and journal row: all or none.
3 · No lost message
A committed message reaches a turn that consumes it while any node owns the partition.
4 · Once in state
A message changes state at most once, including a retried outside command.
5 · Timers
Never delivered early, never lost.
6 · Journal order
Rows present plus rows removed by deletes equal the position; the projector never passes an unwritten position.
7 · Takeover
After a take at e + 1 nothing from e commits, and the new owner sees all that did.
8 · Index matches state
An instance's subscriptions are exactly the waits in its committed state.
9 · Memory matches rows
What an owner holds equals the committed row plus its own uncommitted turn.
10 · Create once
Two messages that would create one instance create one, in any order, across takeovers.
11 · Order
One sender's messages to one target are handled in the order sent.
12 · Jobs
A job's result is applied once; a cancelled job's result never.
13 · Events meet
An event and a wait for its key, both committed in the window, meet exactly once.
14 · One bad instance
A failing instance is parked and its partition goes on; a message is parked for killing processes only after the threshold.
The simulation passes at 200,000 runs of 300 actions and 20,000 of 1,000, also with native instances in the mix; mutation checks break the kernel one rule at a time and each is caught, except two rules that another rule covers, which are caught when that rule is broken too; the store conformance suite runs on the in-memory store, SQLite, PostgreSQL, SQL Server, MySQL and MongoDB (run commands are in the prototype README); a stress test with killed connections holds on all five engines and fails on every one with the fence broken on purpose. The figures and seeds are in the prototype's FINDINGS.md (sections 22 to 24); the engine findings, among them E9 (SQL Server deadlocks with the fence in the batch), in ENGINES.md.
Trade-offs and the chosen defaults
The owner set the order to optimise in: correctness, then database calls per workflow, then latency, then flexibility. The original 12 November defaults are shown below; the partition experiment was amended on 15 November.
Adopted experiment, not implemented: compare 1,024 and 4,096 fixed virtual partitions, provisionally 4,096. Both fence variants retain a gapless ordered journal per virtual partition. Final fence granularity is the owner's decision after measurements; existing diagrams retain the original prototype numbers.
| # | Gain | Price | Lever |
|---|---|---|---|
| 1 | No reads while a workflow runs: state is in memory | Memory per active instance; the whole state is rewritten at each commit | Bound on instances held; large values by reference |
| 2 | One writer per partition: no locks, many instances per commit | Failover takes about one lease; a hot partition has one writer; partition count fixed at creation | Partition count; lease length |
| 3 | An idle owner needs no traffic… | …except an empty fenced commit about three times per lease, per partition | Lease length |
| 4 | Batching under load | A commit waits for the one before it: latency rises a little, throughput a lot | Rows per flush |
| 5 | Balanced: one commit per wait | After a crash, activities since the last commit run again; outside side effects may repeat | Persistence mode; Durable for what must not repeat |
| 6 | Background jobs do not block the instance | It can run more than once; external calls need idempotency keys | None: inherent, as in Temporal |
| 7 | The run viewer costs nothing on the hot path | The run view stays current by reading the journal tail (decided, not built); lists over projected rows lag the projector; its writes still cost calls | History level per workflow |
| 8 | Instances only talk through messages: no races | No code reads another instance's state; a question is a message or a lagging projection | None: it is the design |
| 9 | State is a snapshot: no determinism rules for authors | History is a record, not the source; old state needs upgrade code | State versions, UpgradeState |
| 10 | State is opaque: any model fits | The kernel cannot search inside state; indexes cost writes | Describe; indexed fields per workflow |
| 11 | A message from outside costs little | One extra small commit (mark before running) and a poll; another node's resume pays a hop | Poll interval; notifier |
| 12 | One shared database: simple, transactional across everything | The cluster's ceiling is that database's write throughput; a turn is never free of calls, unlike Zeebe | Later, if needed |
| 13 | Persistence is abstracted: any store | No provider shortcuts in the kernel; each provider needs tuning; MongoDB needs a replica set | Store conformance suite |
| 14 | The actor runtime can be swapped | Runtimes host partitions only; one grain per instance is deferred | Ship one runtime; adapters on demand |
| 15 | A poison message cannot walk the cluster | Suspects are retried one at a time after a takeover; an innocent one is rarely parked | Threshold; trial group size |
| 16 | No HTTP response for lost work | The response waits 2–3 round trips on PostgreSQL for its commit; a resume on another node pays a hop | Release before commit, per route |
| 17 | A clean break: the smallest, fastest design | No migration of existing runs; the native interpreter is rewritten as a model; two cores until the switch | None: accepted 11 and 12 November |
Open and not built
Not built yet
- The Orleans reference adapter and the runtime conformance suite
- Deployment profiles as code; inline execution on every profile
- The graph engine extracted from the Flowchart; spikes for BPMN on Bpmn.Semantics, StateMachine, agents, standalone jobs (Elsa.Jobs as a job actor type, with the worker protocol) and ETL's fan-out pattern
- Short timers and jobs of an Ephemeral instance kept in memory; the stress test with native instances; a three-node MongoDB replica set
- Streaming HTTP responses; a route mapping a client's disconnect to a cancel; the owner's signal to the receiving node
- SQL Server's checked fence as its default
Decided 12 November
The prototype's README and findings track current implementation coverage and its limits. The decisions below record the proposed design.
- Commit fence in the batch by default (2 round trips on PostgreSQL); fence first stays an option
- A pluggable wake signal between nodes; polling remains the guarantee
- Short timers and jobs of an Ephemeral instance stay in memory, up to the route timeout or a few seconds
- Events: subscription and event tables are the one kernel mechanism
- Reply addresses expire at the route's timeout, then the instance is unpinned
- Recreate after delete: generated ids never return; caller keys may after
keep_until - The projector batches each round into set-based writes, off the hot path
- Jobs are counted for poison like messages
- Large values stored by reference; segments only if measurements need them
- The viewer reads projected rows plus the journal tail, so it is always current
- Orleans hosts partitions as grains; no grain per instance, no sweep
- 13 November: SQL Server defaults to the fence as its own statement (E9); the architecture section goes in
docs/architecture/, published at the switch
To answer in the design proposal
- Migrating running instances: a hook per container engine, and versioned activity state
- Data lifecycle: retention, the right to erasure in an append-only journal, encryption at rest
- Fairness, per-tenant quotas, admission control and bounded inboxes
- Fan-out and fan-in over 100,000 children without flooding the parent
- Job workers outside the process, in any language, with leases and heartbeats, built from Elsa.Jobs's worker protocol
- Observability: trace context, stable metrics, operator actions on parked work
- Write volume and bloat: bytes per commit and PostgreSQL dead rows, measured
Still open
- What the store contract says about subscribing twice and removing what is absent
- Tenant and trace headers;
Describefor lists - StateMachine implementation design (#3815); inclusion in both release-one profiles adopted 15 November
- The default join for a Flowchart node with several inbound connections
- BPMN's scope for its first release
Release 1 scope (amended 15 November)
In release 1
- The kernel, the built-in runtime, the Embedded and Cluster profiles
- The same Native model set in both profiles: structured containers, Flowchart and StateMachine
- Synchronous HTTP
- PostgreSQL, SQL Server and SQLite, tuned; MySQL supported, not tuned
- Studio on the new core
- Basic trace context and metrics, retention and delete, admission control, simple instance migration
- Parked-work operator tools: inspect, retry and discard
- Values by reference (decision 10), a release 1 priority
- Agents as native activities, agent calls as background jobs
After release 1
- BPMN and ETL, each proven by a spike now
- Orleans only after proven demand; no automatic product commitment
- MongoDB in 1.1; proven in CI against a replica set from release 1
- The aggregating fan-out pattern, with ETL; release 1 caps a parallel ForEach
- The out-of-process job worker protocol (1.x); its contract designed now
- A dedicated agent execution model
Standalone jobs (Elsa.Jobs on the kernel) are in neither column: their release is decided in the design proposal.
Program #4229 tracks the remaining prototype work and its evidence. The design proposal includes a new ADR, a program plan, the architecture references and a list of ADRs to retire, followed by the Fable review and owner approval. No product code until the design is approved.