Distributed Coordination

#Database as source of truth

Mechanoid prefers a load-on-demand model over in-memory cluster gossip:

• State lives in the EventStore

• Application nodes stay stateless: any node can handle any instance

• Fresh loads see the latest events; no pub/sub membership protocol

Optimistic sequence numbers always detect write conflicts. Distributed locking prevents them.

#LockingStrategy

{
  enum OrderState derives Finite:
    case Pending, Paid, Shipped

  enum OrderEvent derives Finite:
    case Pay, Ship

  import OrderState.*, OrderEvent.*

  val machine = Machine(
    assembly[OrderState, OrderEvent](
      Pending via Pay to Paid,
      Paid via Ship to Shipped,
    )
  )

  val orderId: OrderId = "order-lock-1"

  ZIO
    .scoped {
      for
        fsm   <- FSMRuntime(orderId, machine, Pending)
        _     <- fsm.send(Pay)
        state <- fsm.currentState
      yield state
    }
    .provide(
      InMemoryEventStore.layer[OrderId, OrderState, OrderEvent],
      ZLayer.fromZIO(InMemoryFSMInstanceLock.make[OrderId]),
      TimeoutStrategy.fiber[OrderId],
      LockingStrategy.distributed[OrderId],
    )
    .asDoc
}

Paid

#Lock heartbeat and atomic transitions

withLockAndHeartbeat renews the lock while long work runs (LockHeartbeatConfig: renewalInterval, renewalDuration, jitterFactor, onLockLost).

LockedFSMRuntime.withAtomicTransitions holds one lock across a multi-step sequence so validation → approval style flows stay exclusive. Contention still surfaces as SequenceConflictError under optimistic locking; distributed locking reduces that race by serializing writers per instance id.

Combine durable timeouts + distributed locking for production multi-node setups:

.provide(
  eventStoreLayer,
  timeoutStoreLayer,
  lockServiceLayer,
  TimeoutStrategy.durable[OrderId],
  LockingStrategy.distributed[OrderId],
)

Next: Visualization.