Indexing

Why not alias

Aliases are unique: Alias.of[S].campaign(id) maps to one instance. UniqueAliasError still holds when index rows exist for the same instance. A person with N tickets is the opposite: many instances share one key. Query that with IndexQuery.of[S].assignee(me), not Alias.of.

{
  enum S derives Finite:
    case Open(@alias owner: String, @index assignee: String)
  (for
    index <- InMemoryInstanceIndex.make[String]
    _     <- index.bind(Alias.of[S].owner("p-1"), "unique")
    clash <- index.bind(Alias.of[S].owner("p-1"), "other").either
    _     <- index.bindIndexes(
      Chunk(IndexQuery.of[S].assignee("p-1").toQuery[String].key),
      "t-1",
      meta("Open", t0),
    )
    _ <- index.bindIndexes(
      Chunk(IndexQuery.of[S].assignee("p-1").toQuery[String].key),
      "t-2",
      meta("Open", t1),
    )
    page <- index.find(IndexQuery.of[S].assignee("p-1").limit(8))
  yield (clash.isLeft, page.items.map(_.instanceId).toSet)).asDoc
}

(true,Set(t-2, t-1))

Extractor is a function first

IndexExtractor.apply is the API. Nested payloads write f by hand (the function body may encode keys internally). Query call sites still use IndexQuery.of[S].assignee(me), not a string. The macro walks constructor parameters only, not nested products.

{
  final case class Body(assignee: String, createdAt: Instant, editedAt: Instant)
  enum Ticket derives Finite:
    case Open(body: Body)
    case Archived(body: Body)
  val ext = IndexExtractor[Ticket](
    keys = {
      case Ticket.Open(b)     => Chunk(IndexQuery.of[DocsTicket].assignee(b.assignee).toQuery[String].key)
      case Ticket.Archived(b) => Chunk(IndexQuery.of[DocsTicket].assignee(b.assignee).toQuery[String].key)
    },
    clocksOf = {
      case Ticket.Open(b)     => Some(IndexClocks(b.createdAt, b.editedAt))
      case Ticket.Archived(b) => Some(IndexClocks(b.createdAt, b.editedAt))
    },
  )
  val s = Ticket.Open(Body("p-1", t0, t1))
  ZIO.succeed((ext.indexes(s).map(_.namespace), ext.clocks(s).isDefined))
}

(Chunk(assignee),true)

@alias and @index are members

There is no string parameter on @alias / @index. The namespace is the constructor param name. assigneeId stays assigneeId. Rename the field when you want a different member at the query site (IndexQuery.of[S].assignee(me)).

docsTicketExtractor.indexes(DocsOpen("p-1", "alpha", 3)).map(_.namespace).toSet

Set(assignee, project)

Domain clocks

@indexCreated / @indexUpdated copy Instants from the state onto the covering row. Inbox: .sort(IndexSort.EditedDesc). Activity feed: TouchedDesc. Without domain clocks they agree (edited_at == touched_at).

docsTicketExtractor.clocks(DocsClocked("p-1", t0, t1))

Some(IndexClocks(Some(2020-01-01T00:00:00Z),Some(2020-01-02T00:00:00Z)))

{
  val q = IndexQuery.of[DocsTicket].assignee("p-1")
  (for
    index <- InMemoryInstanceIndex.make[String]
    _     <- index.bindIndexes(
      Chunk(q.toQuery[String].key),
      "t-1",
      IndexMeta("Open", t0, t2, Some(IndexClocks(t0, t1))),
    )
    _ <- index.bindIndexes(
      Chunk(q.toQuery[String].key),
      "t-2",
      IndexMeta("Open", t0, t1, Some(IndexClocks(t0, t2))),
    )
    edited  <- index.find(q.sort(IndexSort.EditedDesc).limit(8))
    touched <- index.find(q.sort(IndexSort.TouchedDesc).limit(8))
  yield (edited.items.map(_.instanceId), touched.items.map(_.instanceId))).asDoc
}

(Chunk(t-2,t-1),Chunk(t-1,t-2))

My Archived assignments

.assignee(me).only(state[Archived]) is equality on the btree suffix (namespace, index_key, state_name). Postgres pushes state_name IN (...). That is not Except, and not a global state key.

{
  val q = IndexQuery.of[DocsTicket].assignee("p-1")
  (for
    index <- InMemoryInstanceIndex.make[String]
    _     <- index.bindIndexes(Chunk(q.toQuery[String].key), "open", meta(openName, t0))
    _     <- index.bindIndexes(Chunk(q.toQuery[String].key), "arch", meta(archivedName, t1))
    page  <- index.find(q.only(state[DocsArchived]).limit(8))
    n     <- index.count(q.only(state[DocsArchived]))
  yield (page.items.map(_.instanceId), n)).asDoc
}

(Chunk(arch),1)

only, all, and except

.only(state[Open], state[InProgress]) or .only(all[Active]) names the leaves. .except(state[Archived]) is correct but walks the other leaves for that key. Prefer only when the inbox must not touch archived rows even as a btree subrange.

{
  val q = IndexQuery.of[DocsTicket].assignee("p-1")
  (for
    index  <- InMemoryInstanceIndex.make[String]
    _      <- index.bindIndexes(Chunk(q.toQuery[String].key), "open", meta(openName, t0))
    _      <- index.bindIndexes(Chunk(q.toQuery[String].key), "doing", meta(doingName, t1))
    _      <- index.bindIndexes(Chunk(q.toQuery[String].key), "arch", meta(archivedName, t2))
    active <- index.find(q.only(all[DocsActive]).limit(8))
    pair   <- index.find(q.only(state[DocsOpen], state[DocsInProgress]).limit(8))
    except <- index.find(q.except(state[DocsArchived]).limit(8))
  yield (
    active.items.map(_.instanceId).toSet,
    pair.items.map(_.instanceId).toSet,
    except.items.map(_.instanceId).toSet,
  )).asDoc
}

(Set(doing, open),Set(doing, open),Set(doing, open))

require AND

.assignee(me).require.project(proj) intersects posting lists. The access path (assignee) owns covering rows and the cursor. Extra equalities are filters. Missing require keys yield an empty page.

(for
  index <- InMemoryInstanceIndex.make[String]
  a = IndexQuery.of[DocsTicket].assignee("p-1")
  _ <- index.bindIndexes(
    Chunk(a.toQuery[String].key, IndexQuery.of[DocsTicket].project("alpha").toQuery[String].key),
    "both",
    meta("Open", t0),
  )
  _    <- index.bindIndexes(Chunk(a.toQuery[String].key), "only-me", meta("Open", t1))
  page <- index.find(a.require.project("alpha").limit(8))
yield page.items.map(_.instanceId)).asDoc

Chunk(both)

Rank

@indexRank covers an Int / Long / Short (widened to Long, default 0). .sort(IndexSort.RankDesc).rankMin(3L) ranges on that column. Equality keys cannot range.

{
  val q = IndexQuery.of[DocsTicket].assignee("p-1")
  (for
    index <- InMemoryInstanceIndex.make[String]
    k = q.toQuery[String].key
    _    <- index.bindIndexes(Chunk(k), "lo", meta("Open", t0, 1L))
    _    <- index.bindIndexes(Chunk(k), "hi", meta("Open", t1, 9L))
    page <- index.find(q.sort(IndexSort.RankDesc).rankMin(3L).limit(8))
  yield (page.items.map(_.instanceId), docsTicketExtractor.rank(DocsOpen("p-1", "alpha", 4)))).asDoc
}

(Chunk(hi),Some(4))

Cursor pagination

Exclusive startAfter (seek after the last row), IndexPage.cursor, hasMore when items.size == limit. No OFFSET. The cursor is (IndexScalar, instanceId).

{
  val q = IndexQuery.of[DocsTicket].assignee("p-1").sort(IndexSort.EditedAsc).limit(1)
  (for
    index <- InMemoryInstanceIndex.make[String]
    k = q.toQuery[String].key
    _  <- index.bindIndexes(Chunk(k), "a", meta("Open", t0))
    _  <- index.bindIndexes(Chunk(k), "b", meta("Open", t1))
    p1 <- index.find(q)
    p2 <- index.find(q.startAfter(p1.cursor.get))
    p3 <- index.find(q.startAfter(p2.cursor.get))
  yield (p1.items.map(_.instanceId), p2.items.map(_.instanceId), p3.items.isEmpty, p1.hasMore)).asDoc
}

(Chunk(a),Chunk(b),true,true)

Birth without send

IndexExtractor.indexes(initial) is bound on reconstruct / apply, not only after append. A cold store with no events still finds the instance.

{
  val machine = Machine(
    assembly[DocsTicket, DocsTicketEvent](
      (state[DocsOpen] via DocsTicketEvent.Tick).to(stay) { (s, _) => s }
    )
  )
  ZIO.scoped {
    for
      store <- InMemoryEventStore.make[TicketId, DocsTicket, DocsTicketEvent]()
      index <- InMemoryInstanceIndex.make[TicketId]
      _     <- ZIO
        .scoped {
          FSMRuntime("t-1", machine, DocsOpen("p-1", "alpha", 1), docsTicketExtractor).unit
        }
        .provide(layers(store, index))
      page <- index.find(IndexQuery.of[DocsTicket].assignee("p-1").limit(8))
    yield page.items.map(_.instanceId)
  }.asDoc
}

Chunk(t-1)

Do not scan snapshots

Hydrate from EventStore only for the instanceIds find returned, and only for fields that are not on Indexed. The store column for a leaf is Finite's simple name; the query API is state[Archived], not that string.

(for
  index <- InMemoryInstanceIndex.make[String]
  _     <- index.bindIndexes(
    Chunk(IndexQuery.of[DocsTicket].assignee("p-1").toQuery[String].key),
    "t-1",
    meta("Open", t0),
  )
  page <- index.find(IndexQuery.of[DocsTicket].assignee("p-1").limit(8))
yield page.items.head).asDoc

Indexed(t-1,Open,2020-01-01T00:00:00Z,2020-01-01T00:00:00Z,2020-01-01T00:00:00Z,2020-01-01T00:00:00Z,0)

Namespace discipline

InstanceIndex is not parameterized by state type. One table per database. Do not share assignee across unrelated machines; pin S with IndexQuery.of[Ticket] vs IndexQuery.of[Doc]. Same rule as Alias.of[S].

(
  IndexQuery.of[DocsTicket].assignee("p-1").toQuery[String].key.namespace,
  IndexQuery.of[IndexDemoTicket].assignee("p-1").toQuery[String].key.namespace,
)

(assignee,assignee)

Backends

Backend	Store	Access path
In-memory	`Map[IndexKey, Map[Id, row]]`	`forward.get(key)` then filter that set
PostgreSQL	`fsm_indexes`	`WHERE namespace AND index_key` plus `state_name IN` for `.only`; rank btree
IndexedDB	`indexes` object store, database version 4	`byKey`; rank on the row; v3 databases upgrade

find does not read fsm_events or fsm_snapshots.

Compile-time

Unknown members and leaves that are not in S fail compilation (report.errorAndAbort). A string literal namespace is rejected. typeCheck is the test, not a runtime lookup.

{
  val unknown = typeCheck("""
          import mechanoid.*
          import mechanoid.persistence.IndexQuery
          enum T derives Finite:
            case Open(@index assignee: String)
          IndexQuery.of[T].typo("x")
        """)
  val badLeaf = typeCheck("""
          import mechanoid.*
          import mechanoid.machine.state
          import mechanoid.persistence.IndexQuery
          enum T derives Finite:
            case Open(@index assignee: String)
          enum Other derives Finite:
            case Archived
          IndexQuery.of[T].assignee("p").only(state[Other.Archived])
        """)
  unknown.zip(badLeaf)
}

(Left(value typo is not a member of mechanoid.persistence.IndexQueryOf[T]{
  val assignee: mechanoid.persistence.IndexNsBind[T]}),Left(Cannot extract Finite leaf from type: <<Internal compiler type dotty.tools.dotc.core.Types$PreviousErrorType@600a6633 does not have a corresponding reflect extractor> does not have a source representation>
type Archived is not a member of object Other))

Live ticket index

The panel below is a ticket inbox. Create indexes the ticket immediately. Number is Alias.of[S].number. Filters are .assignee, .only(state[Archived]), .only(all[Active]), .except, .require.project, rank sort, rankMin, and cursor paging. The list is find, not tickets stashed in the panel.

(for
  index <- InMemoryInstanceIndex.make[String]
  k = IndexQuery.of[IndexDemoTicket].assignee(TicketIndexDemoUi.Me).toQuery[String].key
  _        <- index.bindIndexes(Chunk(k), "t-open", meta(demoOpen, t0, 1L))
  _        <- index.bindIndexes(Chunk(k), "t-arch", meta(demoArch, t1, 2L))
  archived <- index.find(
    TicketIndexDemoUi.queryOf(
      TicketIndexDemoUi.Me,
      TicketIndexDemoUi.Chip.ArchivedOnly,
      None,
      TicketIndexDemoUi.SortPick.Edited,
      None,
      None,
      8,
    )
  )
  page1 <- index.find(
    TicketIndexDemoUi.queryOf(
      TicketIndexDemoUi.Me,
      TicketIndexDemoUi.Chip.All,
      None,
      TicketIndexDemoUi.SortPick.Edited,
      None,
      None,
      1,
    )
  )
  page2 <- index.find(
    TicketIndexDemoUi.queryOf(
      TicketIndexDemoUi.Me,
      TicketIndexDemoUi.Chip.All,
      None,
      TicketIndexDemoUi.SortPick.Edited,
      None,
      page1.cursor,
      1,
    )
  )
yield (
  archived.items.map(_.instanceId),
  (page1.items ++ page2.items).map(_.instanceId).toSet,
)).asDoc

(Chunk(t-arch),Set(t-arch, t-open))

TicketIndexDemo.ui

JVM preview (in-memory). Create a ticket, then filter, page, or look it up by number.

New ticket

Ticket number must be unique. Create puts it in the index immediately.

Number

Assignee

Project

Priority

Inbox

The list is an index query, not tickets kept in this panel.

State

Project

Sort

Min priority

No tickets for this query.

Indexing

#Why not alias

#Extractor is a function first

#@alias and @index are members

#Domain clocks

#My Archived assignments

#only, all, and except

#require AND

#Rank

#Cursor pagination

#Birth without send

#Do not scan snapshots

#Namespace discipline

#Backends

#Compile-time

#Live ticket index