Design draft 5 · 12 November 2026 · prototypes/kernel-model

How the clean-slate Elsa kernel fits together

A walk through the new execution kernel, from the three layers down to the rows a commit writes. The diagrams with step buttons can be played through; every step is also written out beside them. New to the words used here? Start with the companion The kernel in motion, which follows one order workflow through every part, with a visual glossary and a lab for the fence.

Standalone model, not product code Design not yet approved Fable review pending Prototype on main: prototypes/kernel-model; gates green
01

The kernel in one picture

The kernel is three layers and one storage contract. The actor runtime decides where things run. The durable layer makes every guarantee: nothing is lost, nothing applies twice, and a stale node can never write. The actor types on top do the actual work, and a workflow instance is one of them. Click a layer to see what it owns.

Hosts HTTP routes · API · Studio add messages to the inbox, observe what an instance emits add message 3 · Actor types Workflow instance definition · status · parent and children · history Native BPMN ETL Agents your model Correlation key one actor per key optional Durable entity a value, changed by operations IExecutionModel · ITurn 2 · Durable actors Durable inbox Fenced commit Timers Background jobs Events Poison trials Journal Instances in memory IActorRuntime · IScopeHost · IScopeActivation IKernelStore Persistence EF Core PostgreSQL · SQL Server MySQL · SQLite MongoDB 8 (replica set) Later: a replicated log store correct = passes the store suite Planned, not designed: replicated log and local state. The dashed uplink to layer 1 ownership means leadership replaces leases; both IKernelStore and IActorRuntime are involved. Replicated log (Zeebe-like) later · planned, not designed leadership replaces leases 1 · Actor runtime Built-in: partitions Orleans adapter (tests) Your adapter runtime conformance suite (planned)
The dashed box is planned, not designed: replicated log plus local state. Its uplink reaches layer 1 ownership: leadership replaces leases, involving both IKernelStore and IActorRuntime. Virtual partitions need not map one to one to Raft groups. Every guarantee lives in layer 2; layer 3 says what an actor does. Built in the prototype: the built-in partition runtime, the durable layer, the workflow instance and the native model, with seam leaks recorded in DESIGN.md. This picture is not product or later-profile qualification.

1 · Actor runtime

Owns
Placing actors on nodes, running each scope one pass at a time, wake-ups between nodes.
Not
Any guarantee about messages or state.
Swap
Built in: partitions. Integrators: Orleans, Proto.Actor, Akka.NET, Dapr actors.

2 · Durable actors

Owns
The durable inbox; a turn's new state and sent messages committed atomically; the fence against a stale owner; timers; background jobs; events; poison handling; the journal.
Not
Where an actor runs.
Swap
Never. Its storage can be swapped.

3 · Actor types

Owns
What an actor does with its messages. A workflow instance holds what all workflows share and runs an execution model for the rest.
Not
Persistence, placement.
Swap
Extended by adding actor types and execution models.

Persistence provider

Owns
Nine core guarantees: atomic commit, a fence inside it, conditional take, create-only inserts, an order key, reads in key order, reads by key, its own clock, failures told apart. The prototype's store also carries the node heartbeat, start subscriptions, held messages, delete and the projector's rows, each tested by the conformance suite.
Not
Any SQL in the kernel.
Swap
Any store that passes the conformance suite.
Why layer 2 is ours on any runtime. Orleans grain calls are at most once, a grain saves its state on its own, and saving state and sending a message are never one atomic act; when the network splits Orleans can briefly run two activations of one grain. Proto.Actor's messages are not durable. So the runtime only places and wakes; the guarantees come from layer 2 and its fence, which catches a second activation.

The idea in five sentences

  1. The durable layer knows nothing about workflows. It runs durable actors that own a piece of state and react to messages.
  2. An execution model turns an instance's state and incoming messages into new state and outgoing messages. It is the only extension point.
  3. Everything that crosses an instance boundary is a message: a start, a child finishing, a timer, an external event, a cancel, an alteration. No code reads another instance's state.
  4. Under the built-in runtime, actors are grouped into partitions. One node owns a partition at a time and is its only writer, so the hot path needs no reads, no per-instance lease and no read before a turn.
  5. A commit writes everything a turn changed in one transaction, and the owner batches the commits of many instances into one.
The inbox alone says what is owed. Anything the kernel still has to do for an instance after a commit (continue, start a background job, look for events) is a message written in that commit. A new owner needs nothing but the rows. The simulation forced this rule three times.
02

Words used here

Actor
Something with its own state that handles one message batch at a time. A workflow instance is one kind.
Turn
An instance's turn to run: it takes the messages in its inbox, runs as many activities as its persistence mode allows, changes its state and sends messages. Temporal calls this a workflow task. One turn is not one activity.
Pass
The runtime's call into a partition it hosts: it runs the turns that are due and commits them in batches. One pass holds many turns.
Partition
A fixed bucket of instances whose owner is their only writer. The original prototype uses 64 on server databases and 1 on SQLite; existing illustrations use that shape. The adopted 15 November experiment compares 1,024 and 4,096 fixed virtual partitions, provisionally 4,096; both fence variants preserve a gapless journal per virtual partition. Final fence granularity awaits the owner's decision after measurements; this experiment is not implemented.
Epoch
A counter on the partition row, raised every time a node takes the partition. Work started under an older epoch is thrown away.
Position
How many journal rows a partition has written (a turn at the Minimal history level writes none). The commit checks it, so two commits can never land at the same position.
Fence
The conditional update inside every commit: it lands only while the epoch and position are still the ones the owner expects.
Lease
How long the owner may hold the partition without writing (10 s). It decides who should write; the fence decides who can.
Message
A row in the inbox table: target, kind, payload, an order key seq, optionally a due time (a timer) and a start spec (create the target if missing).
Job
A background job: something slow an instance starts during a turn, whose result comes back later as a message while the instance keeps taking other messages.
Journal
One compact row per instance commit, written in the commit: the record of what happened. The viewer's tables are built from it.
Persistence mode
Where a workflow ends its turns: Ephemeral (only at a wait, if ever), Balanced (at each wait), Durable (after each activity). Chosen per workflow.
03

The seam between the runtime and the durable layer

Layer 2 hands the runtime a set of ownership scopes, one per partition, and the runtime decides which node hosts each one. The runtime runs a scope one pass at a time; a pass runs the turns that are due. Layer 2 does not trust it to be right: the fence catches a second activation on another node.

public interface IActorRuntime
{
    ScopeId ScopeOf(ActorId actor);                  // the actor's partition
    ValueTask StartAsync(IScopeHost host, CancellationToken ct);
    ValueTask WakeAsync(ActorId actor, CancellationToken ct);   // a hint: there is work
    ValueTask StopAsync(CancellationToken ct);
}

public interface IScopeHost                       // implemented by layer 2
{
    ValueTask<IScopeActivation?> ActivateAsync(ScopeId scope, CancellationToken ct);  // takes the fence, or declines
}

public interface IScopeActivation : IAsyncDisposable
{
    ValueTask WokenAsync(ActorId actor, CancellationToken ct);
    ValueTask<TimeSpan?> RunAsync(CancellationToken ct);        // one pass: turns until idle; when to call again
}

The prototype was split this way in round 4 stage A.

Every runtime hosts partitions (Orleans: a partition per grain) Node A · owns partition 17 at epoch 8 i1 i2 i3 i4 i5 … One committer: many instances, one transaction 1 fence · 1 poll · 2–3 round trips per flush (PostgreSQL) partition 17: epoch 8 · position 1,204 Server engines: 100 commits cost the round trips of one Dropped 12 November: one grain per instance Grains, placed by Orleans i1own fence rowinbox read on wakecommits alone i2own fence rowinbox read on wakecommits alone i3own fence rowinbox read on wakecommits alone Sweep (layer 2): wakes idle actors that have due work and no live owner a fence per actor each commit 2–3 round trips (PostgreSQL); no batching
Decided 12 November: every runtime hosts whole partitions, and the Orleans adapter hosts each partition as a grain. One conditional update fences many instances and their commits share a transaction. One grain per instance is left out: it loses batching, and since nothing polls an idle grain it needs a sweep. It can come later as a new grain type, with an owner epoch on the instance row. User code that wants to call an instance like a grain gets a reference that forwards to the partition's grain.
04

What is stored

Eight record kinds in the shared-store shape, shown as tables; EF and MongoDB implement equivalent records. The checkpoint holds owner state; the journal records committed history. Their current transaction inserts the recipient inbox with sender state, so this shape needs no separate outbox. The accepted provider-neutral rule records sends atomically with sender state and permits deduplicated later delivery through an outbox when cross-partition transactions are unavailable. Click a table.

These cards use the sketch's proposed record names for the existing shared-store shape. Non-owner queries, including lists and counts, use read models in the proposed design. The round 5 H paper separates that proposal from built behavior. The Storage model reference maps the physical EF tables and MongoDB collections, with lifecycles, writers, indexes and retention costs.

lives in target seq order per partition position n+1 … n+k cursor, its own row fair share waits of meet by key delivered as a message partitionepoch · position · lease nodeid · seen_at projectionpartition · position instancestate · revision · jobs messageseq · target · due · seen journalposition · entries subscriptionkey · instance · wait eventkey · id · keep_until Shared-store shape: no outbox, scheduler queue, timer table, per-run lease, commit marker.
Arrows point from a row to the row it refers to. In this shared-store shape every partition commit updates its fence row. These are proposed names for that shape; DESIGN.md gives the prototype's schema in "Data". A future provider may record sends for later delivery instead of inserting the destination inbox in the sender's transaction.

partition

id, owner, epoch, lease_until, position

The fence. epoch changes only on a take; position counts journal rows. Times are in milliseconds; under the default fence lease_until is written from the owner's clock and compared with database time at a take.

Lifecycle and provider reference ↗

node

id, seen_at

A heartbeat, so nodes can count each other and take a fair share of partitions. Liveness only, never correctness.

Lifecycle and provider reference ↗

instance

id, partition, model, definition, status, sub_status, state, state_version, jobs, revision, parent, root, correlation, tenant, created, updated

state is opaque, owned by the model. jobs lists background jobs in flight. This checkpoint is owner state; the proposed non-owner list/count read path uses read models, not this row directly.

Lifecycle and provider reference ↗

message

seq, id, partition, target, kind, payload, headers, sender, reply_to, start, due, seen, trial, attempts, keep_until, done

The inbox. seq from the database gives the order; the unique id makes adding idempotent; a future due makes it a timer or a retry; start creates the target if missing.

Lifecycle and provider reference ↗

subscription

key, instance, wait, partition, kind

Routes external events to waiting instances (kind wait) or to a definition that starts one (kind start).

Lifecycle and provider reference ↗

event

key, id, payload, keep_until

A published event. By default it reaches only waits committed before it; a wait that asks for kept events also gets events published within their window before it.

Lifecycle and provider reference ↗

journal

partition, position, instance, entries

One row per instance commit, written in that commit, according to the history level (Full by default). At Minimal no row is written.

Lifecycle and provider reference ↗

projection

partition, position

How far the projector has got. Its own row, so the projector never touches the fence row and never slows a commit.

Lifecycle and provider reference ↗

05

Life of a turn

Follow one message from the moment someone adds it until its effects are durable, handed on and visible in the viewer. Use the buttons or click a step below the diagram.

insert, unique id hint poll deliver in seq order turn result fenced commit after commit after commit response to the caller in position order CallerAPI, HTTP, other node Owner of partition 17 · epoch 8 Pollerreads rows not yet seen at this epoch Instance i42, state in memorythe model's turn: consume, send, record, emitreads nothing from the database Committerone transaction per flush, many instances i43, same partitionits message, in memory Live channelHTTP response, stream Database messagethe inbox partitionthe fence: epoch, position instancecurrent state, revision journalone row per commit Projectorjournal into the viewer's tables, later
  1. Add a message.

    Anyone may insert one: the API, an event router, another partition's commit. The unique id makes it idempotent. A message from outside carries keep_until, so a retried command gets a duplicate-key answer instead of a second effect.

  2. Wake the owner.

    The adder signals the owner, in process or through a notifier between nodes. It is only a hint; the poll is the guarantee.

  3. Poll.

    A plain read: rows of the partition not seen at this epoch, not done, due within the horizon, in seq order. No locks and no watermark, so a row whose insert commits late is found next time.

  4. Mark it seen.

    A message that came by poll is not acted on until a fenced commit marks it seen = 8. If this node dies, the next owner knows exactly which messages it was holding. Messages sent inside a partition are written as seen and skip this.

  5. Run the turn.

    The execution model gets the state from memory and the inbox in order. It changes state, sends, subscribes, starts jobs, records journal entries and emits live output. Nothing is read from the database.

  6. Commit.

    The committer takes every finished turn of the partition: fence, instance rows, consumed messages deleted, sent messages inserted, journal rows. One transaction, 2 to 3 round trips on PostgreSQL, measured at 100 instances; rows per flush are capped (500 by default in the prototype); SQLite's calls grow with the rows.

  7. Hand over and release.

    Only after the commit lands: a message to an instance in the same partition goes over in memory, and a held HTTP response is released to the caller. A caller never sees a response for work that was then lost.

  8. Project.

    Later, the projector reads journal rows in position order per partition and writes the viewer's tables. The hot path never waits for it. The run view stays current by reading the journal tail past the projector (decided, not built); lists over projected rows lag it slightly.

06

The commit

The owner of partition p at epoch e, which believes the position is n, commits a batch of instance turns in one transaction, k of which write a journal row. Switch between the two PostgreSQL paths. The list groups possible writes by table. Their order can vary with the operations in the batch.

  1. Fence. UPDATE partition SET position = n + k, lease_until = … WHERE id = p AND epoch = e AND position = n. Also renews the lease, so a busy partition needs no heartbeat.
  2. Exactly one row changed? If not, roll back and reset. This check must come before COMMIT is sent.
  3. Insert or update the instance rows the batch changed, each update WHERE revision = r.
  4. Delete the consumed messages, or mark them done when they carry keep_until. Check the count.
  5. Mark seen = e on messages held in memory and not consumed.
  6. Insert the sent messages. Send derives IDs from the sender, revision and ordinal; repeats are refused. Its fixed-width ordinal orders sends within one revision. The provider must preserve send order across revisions too.
  7. Insert and delete subscriptions.
  8. Insert k journal rows at positions n + 1 … n + k.
  9. COMMIT.

Why not one round trip? A commit sent in the same packet as a fence that matched nothing would still land. The owner has to learn the fence held before it lets COMMIT go.

When a commit fails

What happenedWhat the owner does
Unknown outcome: the connection broke mid-commitReset: drop everything in memory for the partition and take it again. Recovery is the start-up path.
The fence matched nothingReset. Another node owns the partition now, or this node's memory is stale.
Deadlock victim, SQLITE_BUSY (transient)Retry the batch as it is.
Definite error: a value too long, a key violationNo reset. Commit the batch's instances one by one and park the one that fails (faulted, with the error). The rest go on.
07

Ownership and takeover

Leases decide who should write. The fence decides who can. Play through a takeover where the old owner was only paused, the case that breaks most designs.

Why the lease needs a fence. A paused owner or a slow packet can outlive a lease. The commit atomically checks the partition's epoch and journal position, so a superseded owner's durable writes roll back. After a lost answer, reset and a new take change the epoch; the position chiefly orders the journal. Ownership and fencing explains the row lock, conflict retries, round-trip cost and alternatives.
Node A Node B partition 17 time → takeepoch 7 · pos 40 commit 3 turnspos 43, lease renewed Node A paused: GC pause or network split lease runs out takeepoch 8 · pos 43+1 attempt on trial poll: every row seen < 8 wakes, commits at epoch 70 rows: refusedresets, drops memory commitpos 44
  1. A takes the partition.

    The take succeeds only if the partition is free, its lease has expired, or A already owns it. Every take raises the epoch: now 7.

  2. A commits.

    The fence checks epoch 7 and position 40 and moves the position to 43. The same update renews the lease. An idle owner sends an empty fenced commit as its heartbeat.

  3. A stops.

    A long GC pause or a network split. A still believes it owns the partition and still holds instances in memory.

  4. The lease runs out.

    Nobody has written, so nothing is wrong yet. A wrong lease costs availability, never correctness.

  5. B takes the partition.

    Epoch 8, position 43. In the same transaction every message A had on trial gets one more attempt: that is how a message that kills its owner gets counted.

  6. B polls everything.

    Every row with seen < 8 comes back, so B finds all A was holding. B builds memory from rows alone, and hands nothing over in memory until a poll comes back short.

  7. A wakes and tries to commit.

    Its commit is fenced on epoch 7. It matches no row, so nothing lands. A resets and drops its memory. This is the Orleans two-activations case, caught.

  8. B carries on.

    B commits position 43 to 44. No two commits ever landed at the same position, and nothing prepared under epoch 7 got in.

  • Fair share. A node counts the others from the node heartbeats, holds its share of partitions and releases the surplus.
  • Reset is start-up. After any doubt the owner drops its memory and takes again. A successful take obtains the true position: relational stores wait behind the transaction; MongoDB may return a transient conflict and require a retry. A take that changes no row returns nothing.
  • Failover becomes eligible after one lease (10 s by default); contention and recovery reads can add time. The epoch fence prevents a superseded owner from committing.
08

Messages, timers and poison

The inbox is the outbox: a message to another instance is a row inserted in the sender's own transaction, and nothing is relayed afterwards.

Order

A target handles its messages in seq order. A sender's commits are serial, so messages from one sender to one target arrive in the order sent.

Once in state

A state change and the delete of the message that caused it are one transaction. A message changes state exactly once, though a turn's code may run more than once after a crash.

Outside commands

Carry keep_until. The consuming commit marks them done and a sweep deletes them later, so a client retrying a handled command gets a duplicate-key answer, not a second effect. Messages between instances need no window: a repeat cannot exist.

Timers

A message with a future due. The poll returns database time; the owner anchors it and schedules on a monotonic clock. Cancel a timer by deleting its message, whose id the sender knows.

Creating instances

A message may carry a start spec: if its target does not exist, the owner creates it on delivery. Starting a child, starting on an event and signal-with-start are all this one mechanism.

Placement

A start without a caller's key goes into a partition the starting node owns and runs there, with no hop. A keyed start hashes the key to a partition. A child goes into its parent's partition, so a call tree commits together.

Poison

A new owner puts its predecessor's held messages on trial, one at a time by default (an optional larger group halves with every attempt). If that owner is lost too, the next take adds one attempt to exactly the messages on trial, so a poison message ends up alone and a death stops counting against its neighbours. Past the threshold it is parked and its instance faulted, so a message that kills its process cannot walk the cluster.

Edge cases

A message to an instance that does not exist and carries no start spec is dropped and journalled. A message to a finished instance is consumed and dropped. A turn that throws parks its instance at once; retries are the model's business.

09

Execution models

A workflow instance runs an execution model. The model is a plug-in with one job: given the state and the inbox, produce new state and outgoing messages. Everything else is the kernel's.

public interface IExecutionModel
{
    string Key { get; }         // "elsa.native", "bpmn", ...
    int StateVersion { get; }
    object CreateState(StartSpec start);
    object UpgradeState(object state, int from);
    ValueTask TurnAsync(ITurn turn, CancellationToken ct);
    ValueTask<object?> ExecuteJobAsync(IJob job, CancellationToken ct);
    InstanceSummary Describe(object state);
}

The proposed contract. The prototype already uses turn, job and pass; where its signatures still differ, its DESIGN.md lists most of them.

public interface ITurn
{
    InstanceInfo Instance { get; }
    IReadOnlyList<Message> Inbox { get; }
    IAmbient Ambient { get; }   // this activation only
    object State { get; set; }
    string Send(...);  void Recall(id);  InstanceRef Start(spec, placement, detached);
    void StartJob(key, args);  void CancelJob(key);
    void Hold(message);
    void Subscribe(key, wait, keep);  void Unsubscribe(...);
    string Publish(key, payload, keep);  bool Redelivered(message);
    void Record(entry);         // journal
    void Emit(channel, item);   // live, not durable
    void Rebind(definition);  void SetStatus(...);
}

A persistence mode is just where a turn ends

A turn commits at its end and only there. A model that wants a durable boundary ends the turn and sends itself a "continue" message, which the owner hands over in memory. That is all a persistence mode is. Here are ten activities with no wait, measured on PostgreSQL:

With a wait in the middle, Balanced also commits once at the wait. After a crash, Balanced runs again the activities since its last commit: activity code runs at least once, its effect on the workflow's state exactly once.

How the known models sit on the contract

Only Native is built, as a spike; the others are the planned fit.

Native (Elsa)

State is the live activity frames, variables and scheduled work. Slow activities are jobs. Each container brings its own semantics (below). Built as a spike; it passes its own simulation checks.

BPMN

Hosted on Bpmn.Semantics, a pure interpreter: definition, token state and one event in; next state and commands out. A turn feeds it the message; its commands become jobs, timers, subscriptions and sends. State is tokens and scopes. A task is a job, so an interrupting boundary event can cancel it. A call activity is a start. A message that arrives early waits in event.

ETL

State is stage cursors and references to data held elsewhere. Partitions of work are children spread over partitions. Backpressure is a credit protocol between instances.

Agents

State is the conversation checkpoint (a Microsoft Agent Framework thread). A tool call is a job; streamed output is emitted; another agent is another instance; a human approval is a subscription.

Standalone jobs

Elsa.Jobs rebuilt on the kernel: a job is an actor type next to the workflow instance, so a .NET app gets durable jobs without workflows. A workflow's job is the same thing, in the same commit, with no bridge between stores.

Durable entities

State is the entity's value; operations are messages with a reply-to. The reviewers' test of a fifth model: it fits with no change.

Replay-style authoring

Temporal and Durable Task style code, or Restate's journaled awaits, fit as models on the same contract. The kernel imposes neither.

Containers and their semantics

The native model is one activity tree. Each container brings its own semantics, runtime state and kind of diagram, and containers nest freely. The kernel sees none of this; it sees turns. Decided 12 November.

ContainerWhat drives itRuntime stateSemantics
Sequence, If, ForEach, While, ParallelThe previous child completingAn index; a fixed joinBuilt in. The structure says whether more can arrive, so no tokens
FlowchartOutcomes choose the flowsTokens (execution paths) and join arrivals per loop roundGraph engine: today's Elsa 4 Flowchart engine (dead-path joins, BPMN's gateway kinds) lifted into a library for the native model
StateMachineEvents fire guarded transitions; exit, transition and entry actions runThe active configuration of every regionStatechart engine: one event per turn; later events wait with Hold while an action is suspended. A native composite (recommended; #3815's design milestone decides), so it works as a root and nested

The plan: extract the graph engine and port the prototype's Flowchart to it (by the prototype's own reasoning, its All join after a decision does not fire; the first step shows it with a test), a BPMN spike that hosts the Bpmn.Semantics interpreter on the kernel, then a StateMachine spike with nested states and parallel regions. Each step is gated by new simulation invariants and mutation checks, and none may move the cost table.

Background jobs

Instance i42 Background job turn: StartJob("charge") commit: jobs entry + ticket ExecuteJobAsync outside the turn; may run again after a crash a timer or cancel: turns run result as a message applied once, ahead of others A new owner finds the ticket by polling and starts the work again. A cancelled job's result is never applied. A result is handled before other messages, so a model never sees a result after its own cancel.

Models compose through messages only. A parent of any model starts a child of any model and gets a message when it finishes. A cancel goes to the children the parent itself started, so admitting a child and cancelling the parent cannot race. The kernel owns suspend and resume, terminating a tree, tenant checks on sends and trace headers.

10

Events and correlation

An external event (an approval, a payment notice, a BPMN message) must reach every instance waiting for its key, even when the event and the wait are committed at nearly the same moment. Each side checks for the other after its own commit.

Waiting instance Publisher commit: subscription row + "read events for k" read the event table for k (the owed read) deliver what is there; next commit clears the debt commit: insert the event read the subscriptions for k add one delivery message per match each sees the other Both sides derive the same delivery id, ev:{event}:{instance}:{wait}, so the double is refused.
Default since the owner's decision of 12 November: events are not kept. Only a wait that asks for kept events (BPMN message correlation) pays for the owed read; the publisher still inserts the event row.

The alternative: correlation as an ordinary instance per key

Built as a 50-line model with no kernel change: subscribing and publishing are messages to the key's instance, which holds the waits and the kept events. It passed every run. It costs a hop, measured on PostgreSQL for one waiter, one event:

VersionCommitsRound trips
Tables (subscription, event, owed read)417 (13)
Key instance in the waiter's partition621 (15)
Key instance in another partition933 (24)

In brackets: with the fence in the batch. Decided 12 November: the tables stay the kernel's one mechanism; the instance per key remains a sample model.

11

Synchronous HTTP

A workflow handles a request and writes the response itself. The response travels on the live Emit channel, not as a durable message: if it cannot reach the caller, the caller gets an error and the workflow is unaffected. Pick a case.

request response commit, then release resume message + reply address poll commit reply, not durable response 503 ClientHTTP request Node A · received the request Route table, in memory New instance, runs inlinerequest as Ambient, in a partition A owns Request held openreply address: node A + request id Node B · owns i42 i42, waitingstate in memory Databasemessage, instance Node B lost
  1. The request arrives.

    Any node can take it. The route table is built when definitions are published, so finding the workflow reads nothing.

  2. It runs inline.

    The node places the new instance in a partition it owns and runs it in the request, with the request as Ambient. No hop.

  3. Respond and finish.

    The workflow writes the response and ends without waiting. In Ephemeral mode nothing is committed: zero database calls.

  • Waiting before responding (a background job, a short timer): the instance stays in memory on the receiving node and the request waits, up to the route's timeout.
  • Client disconnect and timeout end the observation. They do not cancel the workflow, unless the route maps them to a cancel message.
  • Streaming (chunked responses, server-sent events, an agent's tokens) uses the same channel. Large bodies stay out of state; the workflow holds a reference.
12

Deployment profiles

A deployment profile combines an actor runtime and a persistence provider; persistence mode remains the workflow author's choice. Release 1 adopts Embedded and Cluster with the same Native structured containers, Flowchart and StateMachine. Orleans requires proven demand. Log is later, planned, not designed; equivalent behavior and synchronous HTTP remain requirements for that future design. Pick a profile.

Persistence mode

Chosen per workflow by its author. Gives the workflow its guarantees.

EphemeralBalancedDurable

Actor runtime

Layer 1. Where actors run.

Built-in, one node, no transport Built-in, partitions shared over nodes Orleans adapter Planned: replicated-log leadership

Persistence provider

Behind IKernelStore. What persists the data.

SQLite (1 partition) PostgreSQL SQL Server (RCSI on) MySQL MongoDB 8 (replica set) Planned: replicated log (Zeebe-like)

Log profile: planned, not designed. What each kernel concept becomes with a replicated log (Zeebe-like):
Kernel conceptWith a replicated log
Commita fenced batch appended to the partition's Raft-replicated log, acknowledged after consensus commitment
Fence and epochRaft term-based authority, enforced by the commit protocol
Lease and takeleader election
Journalthe log itself
Inbox and pollinglog entries plus local state; no inbox-table polling
Projectoran exporter that reads committed log records past its position
It touches two seams: IKernelStore and ownership in IActorRuntime. The round 5 H paper proposes four contracts before code. The accepted send boundary records sends with sender state and permits deduplicated later delivery; logical per-virtual-partition journal positions remain distinct from physical log indexes.
Inline execution is chosen per workflow or per route: a run that a request starts without a caller's key executes in that request until its first wait, under any persistence mode; a keyed start owned by another node answers over a reply address. Inline plus Ephemeral costs no database calls and no hop. Every profile supports it, because every node owns partitions to place a new instance in.
13

What it costs

EF Core commands per workflow run on SQLite, for three systems in one unit: Elsa 3 (3.10.1), Elsa 4 today (main) and the new kernel's native model. Hover a row for the three figures. On PostgreSQL the kernel's Balanced commit is 2 round trips with the fence in the batch, 3 with the checked fence (NATIVE.md section 5).

Elsa 3 (3.10.1)Elsa 4 today (main)New kernel (prototype)
What is counted: the run's own EF Core commands on SQLite; commands, not time. Elsa 3 (from evidence/elsa3-cost/RECEIPT.md, workflow-flow column): its saves are separate writes with no single transaction, background queue polls are excluded, and the count is largely history (25 of 29, 11 of 15, 12 of 25 in the default mode). Elsa 4 (Essentials/tests/Activities/Http/IntegrationTests/CommandBudgets/ef-command-budgets.json): the gate counts one database context, where Elsa 3's count covers both of its contexts. Kernel (prototypes/kernel-model/NATIVE.md section 5, the SQLite commands in brackets): the history projector is not in its bars (about 4 to 5 SQLite commands per round), so for fork-join the kernel is about level with Elsa 3 once that is added. The kernel is a prototype, not product code.
EngineBalanced run on the ownerNotes
PostgreSQL3 (2)Round trips; in brackets the fence in the batch.
SQL Server4 (3)Needs read-committed snapshot isolation.
MySQL7 (6)Oracle's EF provider; upserts and INSERT IGNORE.
MongoDB 83 (2)Needs a replica set for transactions; the store assigns the order key.
SQLite5Calls into the engine, where EF sends one command per row. This counts BEGIN and COMMIT too, so it is 5 here against the chart's 3 commands.

This table counts a different unit from the chart above: round trips for the server databases, and engine calls (BEGIN and COMMIT included) for SQLite. Measured by the prototype's own cost runs, as reported in its findings; not re-measured for this record.

14

How it is tested

The model is attacked by a deterministic simulation: several nodes against one store, one random action at a time (takes, polls, turns, commits, crashes, a paused owner that wakes with stale memory, commits whose answer is lost, retried commands, clocks that disagree). After every action it checks these invariants. A failing seed replays exactly.

1 · One writer

No two commits land at the same position; nothing prepared under an old epoch lands.

2 · Atomic turn

State, consumed and sent messages, subscriptions and journal row: all or none.

3 · No lost message

A committed message reaches a turn that consumes it while any node owns the partition.

4 · Once in state

A message changes state at most once, including a retried outside command.

5 · Timers

Never delivered early, never lost.

6 · Journal order

Rows present plus rows removed by deletes equal the position; the projector never passes an unwritten position.

7 · Takeover

After a take at e + 1 nothing from e commits, and the new owner sees all that did.

8 · Index matches state

An instance's subscriptions are exactly the waits in its committed state.

9 · Memory matches rows

What an owner holds equals the committed row plus its own uncommitted turn.

10 · Create once

Two messages that would create one instance create one, in any order, across takeovers.

11 · Order

One sender's messages to one target are handled in the order sent.

12 · Jobs

A job's result is applied once; a cancelled job's result never.

13 · Events meet

An event and a wait for its key, both committed in the window, meet exactly once.

14 · One bad instance

A failing instance is parked and its partition goes on; a message is parked for killing processes only after the threshold.

The simulation passes at 200,000 runs of 300 actions and 20,000 of 1,000, also with native instances in the mix; mutation checks break the kernel one rule at a time and each is caught, except two rules that another rule covers, which are caught when that rule is broken too; the store conformance suite runs on the in-memory store, SQLite, PostgreSQL, SQL Server, MySQL and MongoDB (run commands are in the prototype README); a stress test with killed connections holds on all five engines and fails on every one with the fence broken on purpose. The figures and seeds are in the prototype's FINDINGS.md (sections 22 to 24); the engine findings, among them E9 (SQL Server deadlocks with the fence in the batch), in ENGINES.md.

15

Trade-offs and the chosen defaults

The owner set the order to optimise in: correctness, then database calls per workflow, then latency, then flexibility. The original 12 November defaults are shown below; the partition experiment was amended on 15 November.

Side effects at least once in Balanced History: Full Lease: 10 s Original prototype: 64 server partitions, 1 on SQLite HTTP response after commit No reads across instances Events not kept unless asked

Adopted experiment, not implemented: compare 1,024 and 4,096 fixed virtual partitions, provisionally 4,096. Both fence variants retain a gapless ordered journal per virtual partition. Final fence granularity is the owner's decision after measurements; existing diagrams retain the original prototype numbers.

#GainPriceLever
1No reads while a workflow runs: state is in memoryMemory per active instance; the whole state is rewritten at each commitBound on instances held; large values by reference
2One writer per partition: no locks, many instances per commitFailover takes about one lease; a hot partition has one writer; partition count fixed at creationPartition count; lease length
3An idle owner needs no traffic……except an empty fenced commit about three times per lease, per partitionLease length
4Batching under loadA commit waits for the one before it: latency rises a little, throughput a lotRows per flush
5Balanced: one commit per waitAfter a crash, activities since the last commit run again; outside side effects may repeatPersistence mode; Durable for what must not repeat
6Background jobs do not block the instanceIt can run more than once; external calls need idempotency keysNone: inherent, as in Temporal
7The run viewer costs nothing on the hot pathThe run view stays current by reading the journal tail (decided, not built); lists over projected rows lag the projector; its writes still cost callsHistory level per workflow
8Instances only talk through messages: no racesNo code reads another instance's state; a question is a message or a lagging projectionNone: it is the design
9State is a snapshot: no determinism rules for authorsHistory is a record, not the source; old state needs upgrade codeState versions, UpgradeState
10State is opaque: any model fitsThe kernel cannot search inside state; indexes cost writesDescribe; indexed fields per workflow
11A message from outside costs littleOne extra small commit (mark before running) and a poll; another node's resume pays a hopPoll interval; notifier
12One shared database: simple, transactional across everythingThe cluster's ceiling is that database's write throughput; a turn is never free of calls, unlike ZeebeLater, if needed
13Persistence is abstracted: any storeNo provider shortcuts in the kernel; each provider needs tuning; MongoDB needs a replica setStore conformance suite
14The actor runtime can be swappedRuntimes host partitions only; one grain per instance is deferredShip one runtime; adapters on demand
15A poison message cannot walk the clusterSuspects are retried one at a time after a takeover; an innocent one is rarely parkedThreshold; trial group size
16No HTTP response for lost workThe response waits 2–3 round trips on PostgreSQL for its commit; a resume on another node pays a hopRelease before commit, per route
17A clean break: the smallest, fastest designNo migration of existing runs; the native interpreter is rewritten as a model; two cores until the switchNone: accepted 11 and 12 November
16

Open and not built

Not built yet

  • The Orleans reference adapter and the runtime conformance suite
  • Deployment profiles as code; inline execution on every profile
  • The graph engine extracted from the Flowchart; spikes for BPMN on Bpmn.Semantics, StateMachine, agents, standalone jobs (Elsa.Jobs as a job actor type, with the worker protocol) and ETL's fan-out pattern
  • Short timers and jobs of an Ephemeral instance kept in memory; the stress test with native instances; a three-node MongoDB replica set
  • Streaming HTTP responses; a route mapping a client's disconnect to a cancel; the owner's signal to the receiving node
  • SQL Server's checked fence as its default

Decided 12 November

The prototype's README and findings track current implementation coverage and its limits. The decisions below record the proposed design.

  • Commit fence in the batch by default (2 round trips on PostgreSQL); fence first stays an option
  • A pluggable wake signal between nodes; polling remains the guarantee
  • Short timers and jobs of an Ephemeral instance stay in memory, up to the route timeout or a few seconds
  • Events: subscription and event tables are the one kernel mechanism
  • Reply addresses expire at the route's timeout, then the instance is unpinned
  • Recreate after delete: generated ids never return; caller keys may after keep_until
  • The projector batches each round into set-based writes, off the hot path
  • Jobs are counted for poison like messages
  • Large values stored by reference; segments only if measurements need them
  • The viewer reads projected rows plus the journal tail, so it is always current
  • Orleans hosts partitions as grains; no grain per instance, no sweep
  • 13 November: SQL Server defaults to the fence as its own statement (E9); the architecture section goes in docs/architecture/, published at the switch

To answer in the design proposal

  • Migrating running instances: a hook per container engine, and versioned activity state
  • Data lifecycle: retention, the right to erasure in an append-only journal, encryption at rest
  • Fairness, per-tenant quotas, admission control and bounded inboxes
  • Fan-out and fan-in over 100,000 children without flooding the parent
  • Job workers outside the process, in any language, with leases and heartbeats, built from Elsa.Jobs's worker protocol
  • Observability: trace context, stable metrics, operator actions on parked work
  • Write volume and bloat: bytes per commit and PostgreSQL dead rows, measured

Still open

  • What the store contract says about subscribing twice and removing what is absent
  • Tenant and trace headers; Describe for lists
  • StateMachine implementation design (#3815); inclusion in both release-one profiles adopted 15 November
  • The default join for a Flowchart node with several inbound connections
  • BPMN's scope for its first release

Release 1 scope (amended 15 November)

In release 1

  • The kernel, the built-in runtime, the Embedded and Cluster profiles
  • The same Native model set in both profiles: structured containers, Flowchart and StateMachine
  • Synchronous HTTP
  • PostgreSQL, SQL Server and SQLite, tuned; MySQL supported, not tuned
  • Studio on the new core
  • Basic trace context and metrics, retention and delete, admission control, simple instance migration
  • Parked-work operator tools: inspect, retry and discard
  • Values by reference (decision 10), a release 1 priority
  • Agents as native activities, agent calls as background jobs

After release 1

  • BPMN and ETL, each proven by a spike now
  • Orleans only after proven demand; no automatic product commitment
  • MongoDB in 1.1; proven in CI against a replica set from release 1
  • The aggregating fan-out pattern, with ETL; release 1 caps a parallel ForEach
  • The out-of-process job worker protocol (1.x); its contract designed now
  • A dedicated agent execution model

Standalone jobs (Elsa.Jobs on the kernel) are in neither column: their release is decided in the design proposal.

Program #4229 tracks the remaining prototype work and its evidence. The design proposal includes a new ADR, a program plan, the architecture references and a list of ADRs to retire, followed by the Fable review and owner approval. No product code until the design is approved.