Two users are offline. Both add content to the same empty note. They come back online, sync finishes, and one user's edits seem to disappear.

There is no error, and the data is not actually gone from history. But note.get("body") can only return one Text container. The other container was created concurrently and still exists in history, but it is no longer visible in the current document state. From the application's point of view, this looks like data loss.

This is a classic problem in JSON-like CRDTs. Users have run into versions of it in the Loro, Yjs, and Automerge communities. The Appendix has short scripts that reproduce it in all three.

Loro now solves this with Mergeable Containers. They make a child container's identity come from its logical position in the Map, not from the ID of the operation that happened to create it.

Special thanks to Alexis Williams from Synapdeck for the substantial implementation work and design discussion behind this feature.

From the user's point of view, the API change is small. Instead of creating an on-demand child container like this:

// Peer A
doc.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "A");

// Peer B, offline
doc.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "B");

// after sync: only one List is visible at "2026-06-08"

you can use a mergeable child:

// Peer A
doc.getMap("days").ensureMergeableList("2026-06-08").insert(0, "A");

// Peer B, offline
doc.getMap("days").ensureMergeableList("2026-06-08").insert(0, "B");

// after sync: both peers edit the same List

As a rule of thumb, use ensureMergeable* when a child container should be identified by its logical position:

map.ensureMergeableText(key);
map.ensureMergeableMap(key);
map.ensureMergeableList(key);
map.ensureMergeableMovableList(key);
map.ensureMergeableTree(key);
map.ensureMergeableCounter(key);

Use them for fields that should behave like one shared child container for everyone: one shared Text, one shared List, one shared Map, and so on. It should not matter which peer creates that child first. The rest of this post walks through why the problem exists and how the new encoding works.

Why This Happens

CRDTs are usually good at cases like "multiple users editing the same text at the same time" or "multiple users inserting into the same list concurrently." This issue happens one layer earlier: before the peers can edit the same List, Text, or Map, they first need to agree on which child container that key refers to.

Before Mergeable Containers, the recommended workaround was to initialize all required child containers as soon as the parent LoroMap was created. For example, if every note always needs a body text, creating that body together with the note avoids the first-creation race.

That workaround is useful, but it has limits. Some applications cannot know every child container ahead of time. A schema migration may add a new child container to existing documents. A calendar-like document may create child containers by date. A dynamic index may create one child container per user-defined key. In these cases, on-demand creation is natural, and concurrent first creation is hard to avoid.

The root cause is the way regular child Container IDs are represented. A normal child Container ID includes the OpID that created it. Concurrent first creation therefore creates different Container IDs, and the Map conflict-resolution rule decides which one is visible.

The issue is not that List insertion cannot merge. Once both peers are editing the same List, List edits merge normally. The issue is that the two peers created two different Lists at the same Map key.

Why Root Containers Are Naturally Mergeable

In Loro and Yjs, top-level Root Containers are usually accessed by name:

doc.getMap("state");
doc.getText("content");

Here, "state" or "content" is already a stable identity. It does not depend on which peer created it or which operation created it. As long as multiple peers access the same root name, they naturally refer to the same logical Container.

Automerge has a different object identity model, so this root-container comparison is specifically about Loro and Yjs. The broader issue is still similar: when composite values are created concurrently at the same key, the system needs a rule for which object identity becomes visible.

Regular child Containers are different. Their identity is tied to the operation that created them, so two concurrent "first creations" become two different objects.

Mergeable Containers bring the useful part of Root Container identity to selected child Containers: the child identity comes from a deterministic name, not from the creation operation.

API: Explicitly Ensuring a Mergeable Child

This feature does not change the existing setContainer / insertContainer behavior. It adds explicit ensureMergeable* APIs for the mergeable case. In Rust, the same methods use snake case:

map.ensure_mergeable_text("body")?;
map.ensure_mergeable_map("profile")?;

The word ensure is intentional. It returns the child and, if needed, writes the marker that makes it visible at that key. Calling the same method again for the same type is idempotent.

If the key already holds a regular scalar value or a regular child Container, the API returns an error instead of silently overwriting it.

One subtle case is type changes. If one peer asks for a mergeable Text at "field" while another peer asks for a mergeable Map at the same key, Loro still needs one visible value at that key. The Map's normal conflict rule decides which type is visible. The non-visible mergeable child's state is still preserved under its deterministic ID, so switching back to that type can resurface it later.

Core Design: Deterministic CID + Map Slot Marker

Mergeable Containers have two separate layers of representation:

1. The child Container ID derived from the parent Container ID, key, and type. This decides whether peers address the same CRDT object.
2. The parent Map slot. This decides whether that object is currently visible at a key, and which mergeable child type is active there.

Keeping these two layers separate makes the behavior easier to reason about.

1. CID: A Synthetic Root Container ID

A Mergeable Container uses a synthetic ContainerID::Root under an internal namespace. User-created root names cannot use this prefix, so ordinary roots cannot collide with mergeable CIDs:

🤝:<payload>

The payload is derived from the parent Map and the key. The Container type stays in ContainerID::Root.container_type, just like ordinary Root Containers. This lets all peers derive the same child ID without using the creation OpID.

The current encoding keeps nested mergeable Map IDs linear in the logical path length. This change was made before release to avoid recursive CID growth for deeply nested mergeable maps.

payload = base-parent ">" key-1 ">" key-2 ...

$<escaped-root-name>
@<peer-base36>:<counter-base36>

Root map "state", key "note-1", child map:
🤝:$state>note-1        type = Map

Nested key "body" under that mergeable map, child text:
🤝:$state>note-1>body   type = Text

parent = Root("🤝:$state>note-1", Map)
key = "body"
type = Text

2. Map Slot: A Binary Marker Controls Visibility

A deterministic CID alone is not enough because Loro has multiple Container types. If one peer calls ensureMergeableText("field") while another peer concurrently calls ensureMergeableMap("field"), both deterministic child CIDs can exist. The parent Map still needs to decide which type is currently visible at "field". That decision needs to be deterministic and reversible: switching the visible type should not destroy the state of the other mergeable child.

So Loro stores a small activation marker in the parent Map slot. Its meaning is:

At this key of this parent Map, activate a mergeable child of this type.

When a new Loro client reads the slot, it uses the current parent id + key + kind to derive the deterministic mergeable CID, then presents it through the public API as a normal Container:

const body = map.get("body");
// body is a LoroText, not the internal binary marker

When the key is deleted, only the marker is removed. The mergeable child state is not immediately destroyed, because the parent slot controls visibility rather than the child's stored history. Calling this again:

map.ensureMergeableText("body");

resurfaces the same deterministic Text Container.

The marker is also bound to its exact parent, key, and type. That keeps it from accidentally activating a mergeable child if the same binary value is copied somewhere else.

MAGIC[4] + KIND[1] + DIGEST[3]

{ "__loro_mergeable_container__": "Text" }

"__loro_mergeable_text__"

What This Solves for Users

Mergeable Containers are especially useful when eager initialization is not practical.

For example, suppose an application stores one child List per date:

const days = doc.getMap("days");
const entries = days.ensureMergeableList("2026-06-08");
entries.insert(0, "meeting notes");

Or suppose a schema migration lazily adds a new child Map to existing records:

const record = doc.getMap("records").ensureMergeableMap(recordId);
const metadata = record.ensureMergeableMap("metadata_v2");
metadata.set("migrated", true);

In both cases, the child container identity no longer depends on which peer created it first. It depends on the logical position in the document structure.

This makes Mergeable Containers especially useful for:

• date-keyed child lists or maps
• schema migrations that add new child containers lazily
• dynamic per-user or per-entity subdocuments
• revision counters
• settings maps whose keys are discovered over time

Cost and Compatibility

Mergeable Containers have some metadata cost. Their CIDs carry logical path information, so deeper paths and longer keys produce larger IDs. PR #1002 changed the encoding so nested mergeable Map IDs grow linearly instead of recursively, but very deep mergeable Map chains are still better to avoid.

The compatibility story is intentionally conservative:

• Existing setContainer / insertContainer behavior is unchanged.
• Existing documents can be read normally by new versions.
• Mergeable Containers are introduced through new APIs, without changing existing method signatures.
• Older clients that do not understand this feature see the parent slot marker as an ordinary binary value, not as a fake child Container edge. They can preserve and sync the data, but they will not display the mergeable child with the new semantics.
• User-created root names that start with the internal 🤝: prefix are rejected by Loro's root-name validator, so they cannot collide with mergeable CIDs.

Summary

Mergeable Containers are for child Containers whose identity should come from their logical position, not from whichever peer created them first.

Use ensureMergeable* when:

• the key is dynamic or lazily created
• different peers may initialize the same child while offline
• the child should behave like one shared Text, List, Map, Tree, or Counter
• deleting the key should hide the child without treating its internal history as immediately destroyed

Keep using setContainer / insertContainer when:

• each creation should produce a distinct child object
• the parent slot should point to exactly the Container created by that operation
• you are modeling replacement rather than shared initialization

The short version: if two peers creating the same child at the same Map key should mean "we both found the same child," use a Mergeable Container.

References:

• Loro background: issue #759
• Loro implementation: PR #991, PR #1002
• Related Yjs discussions: complex diagram page, losing data, nested Y.Map
• Related Automerge discussions: #528: failing merge for text values is the closest match; #526: conflict resolution for replaced arrays and objects is useful background on object identity and conflict handling; the historical automerge-classic #4 also covers concurrently created objects under the same key.

Appendix: Runnable Reproductions

The snippets below are self-contained and run directly on Node (tested on Node 24; any Node 18+ with ESM works). Install the three libraries once:

npm install loro-crdt@^1.13 yjs @automerge/automerge

Save each block as a .mjs file and run it with node file.mjs. They all model the same scenario from this post: two offline peers concurrently create a child container under the same Map key, then sync. The Loro example also shows the ensureMergeable* fix.

Loro — the bug, and the fix

// loro.mjs  —  node loro.mjs
// In plain Node import from "loro-crdt/nodejs"; the bare "loro-crdt" entry
// targets a bundler. With Vite/webpack, import from "loro-crdt" instead.

function sync(a, b) {
  const va = a.export({ mode: "update" });
  const vb = b.export({ mode: "update" });
  a.import(vb);
  b.import(va);
}

// 1. The bug: concurrent setContainer at the same key
{
  const a = new LoroDoc();
  const b = new LoroDoc();
  a.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "A");
  b.getMap("days").setContainer("2026-06-08", new LoroList()).insert(0, "B");
  sync(a, b);
  console.log(
    "setContainer ->",
    JSON.stringify(a.getMap("days").get("2026-06-08").toArray()),
  );
  // -> ["A"] or ["B"], never both: only one peer's List survives.
  // (which one wins depends on the randomly-assigned peer IDs)
}

// 2. The fix: concurrent ensureMergeableList at the same key
{
  const a = new LoroDoc();
  const b = new LoroDoc();
  a.getMap("days").ensureMergeableList("2026-06-08").insert(0, "A");
  b.getMap("days").ensureMergeableList("2026-06-08").insert(0, "B");
  sync(a, b);
  console.log(
    "ensureMergeable ->",
    JSON.stringify(a.getMap("days").get("2026-06-08").toArray()),
  );
  // -> both entries, e.g. ["A","B"] (order may vary): both peers share one List.
}

Yjs — the same problem

// yjs.mjs  —  node yjs.mjs

const a = new Y.Doc();
const b = new Y.Doc();

// Peer A and Peer B each create a Y.Array at the same key, offline.
{
  const l = new Y.Array();
  a.getMap("days").set("2026-06-08", l);
  l.insert(0, ["A"]);
}
{
  const l = new Y.Array();
  b.getMap("days").set("2026-06-08", l);
  l.insert(0, ["B"]);
}

// Sync both ways.
Y.applyUpdate(a, Y.encodeStateAsUpdate(b));
Y.applyUpdate(b, Y.encodeStateAsUpdate(a));

console.log(
  "yjs ->",
  JSON.stringify(a.getMap("days").get("2026-06-08").toArray()),
);
// -> ["A"] or ["B"], never both: one peer's child Y.Array wins, the other is dropped.

Automerge — the same problem

// automerge.mjs  —  node automerge.mjs

let base = A.from({ days: {} });
let a = A.clone(base);
let b = A.clone(base);

// Peer A and Peer B each create a list at the same key, offline.
a = A.change(a, (d) => {
  d.days["2026-06-08"] = ["A"];
});
b = A.change(b, (d) => {
  d.days["2026-06-08"] = ["B"];
});

let merged = A.merge(A.clone(a), b);
console.log("automerge visible ->", JSON.stringify(merged.days["2026-06-08"]));
// -> ["A"] or ["B"], never both: one list wins.
console.log(
  "automerge conflicts ->",
  JSON.stringify(A.getConflicts(merged.days, "2026-06-08")),
);
// -> both lists keyed by op id: the losing list is retained but hidden,
//    reachable only via getConflicts().

// Control: when the child is created ONCE up front, concurrent edits merge.
let shared = A.from({ days: { "2026-06-08": [] } });
let c = A.clone(shared),
  d = A.clone(shared);
c = A.change(c, (x) => {
  x.days["2026-06-08"].push("A");
});
d = A.change(d, (x) => {
  x.days["2026-06-08"].push("B");
});
let ok = A.merge(A.clone(c), d);
console.log("automerge pre-created ->", JSON.stringify(ok.days["2026-06-08"]));
// -> ["A","B"] (order may vary): both survive — this is the eager-init workaround.

Note one difference worth calling out: in Automerge the losing child is retained and can be recovered through getConflicts(), while Yjs overwrites the map key and drops the losing child outright. Either way, from the application's point of view it looks like data loss — which is exactly what Mergeable Containers avoid.

The open-source Loro Protocol project includes the loro-websocket package, the adaptor suite in loro-adaptors, and matching Rust client and server implementations that all interoperate on the same wire format.

The Loro Protocol is a wire protocol designed for real-time CRDT synchronization. Learn about the design in detail here.

It efficiently runs multiple, independent "rooms" over a single WebSocket connection.

This allows you to synchronize your application state, such as a Loro document, ephemeral cursor positions, and end-to-end encrypted documents, over one connection. It is also compatible with Yjs.

Quick Start: Server & Client Example

The protocol is implemented by the loro-websocket client and a minimal SimpleServer for testing. These components are bridged to your CRDT state using loro-adaptors.

Server

For development, you can run the SimpleServer (from loro-websocket) in a Node.js environment.

// server.ts

const server = new SimpleServer({
  port: 8787,
  // SimpleServer accepts hooks for authentication and data persistence:
  // authenticate: async (roomId, crdt, auth) => { ... },
  // onLoadDocument: async (roomId, crdt) => { ... },
  // onSaveDocument: async (roomId, crdt, data) => { ... },
});

server.start().then(() => {
  console.log("SimpleServer listening on ws://localhost:8787");
});

Client

On the client side, you connect once and then join multiple rooms using different adaptors.

// client.ts

// 1. Create and connect the client
const client = new LoroWebsocketClient({ url: "ws://localhost:8787" });
await client.waitConnected();
console.log("Client connected!");

// --- Room 1: A Loro Document (%LOR) ---
const docAdaptor = new LoroAdaptor();
const docRoom = await client.join({
  roomId: "doc:123",
  crdtAdaptor: docAdaptor,
});

// Local edits are now automatically synced
const text = docAdaptor.getDoc().getText("content");
text.insert(0, "Hello, Loro!");
docAdaptor.getDoc().commit();

// --- Room 2: Ephemeral Presence (%EPH) on the SAME socket ---
const ephAdaptor = new LoroEphemeralAdaptor();
const presenceRoom = await client.join({
  roomId: "doc:123", // Can be the same room ID, but different magic bytes
  crdtAdaptor: ephAdaptor,
});

// Ephemeral state syncs, but is not persisted by the server
ephAdaptor.getStore().set("cursor", { x: 100, y: 100 });

Features

Multiplexing

Each binary message is prefixed with four magic bytes that identify the data type, followed by the roomId. This structure allows the server to route messages to the correct handler. A single client can join:

• %LOR (Loro Document)
• %EPH (Loro Ephemeral Store, for cursors and presence)
• %ELO (End-to-End Encrypted Loro Document)
• %YJS and %YAW (for Yjs Document and Awareness interoperability)

All traffic runs on the same socket.

Compatibility

The Loro Protocol is designed to accommodate environments like Cloudflare:

• Fragmentation: Large updates are automatically split into fragments under 256 KiB and reassembled by the receiver. This addresses platforms that enforce WebSocket message size limits.
• Application-level keepalive: The protocol defines simple "ping" and "pong" text frames. These bypass the binary envelope and allow the client to check connection liveness, which is useful in browser or serverless environments where transport-level TCP keepalives are not exposed.

This repository also ships Rust clients and servers that mirror the TypeScript packages.

Experimental E2E Encryption

End-to-end encrypted Loro is included in loro-protocol, but the feature is currently experimental: expect wire formats and key-management APIs to change, and do not rely on it for production-grade security audits yet. When paired with EloLoroAdaptor on the client, the server relays encrypted records without decrypting them.

Status and Licensing

The Loro Protocol is mostly stable. We welcome community feedback and contributions, especially regarding use cases that are difficult to satisfy with the current design.

All the packages in inside https://github.com/loro-dev/protocol are open-sourced under the permissive MIT license.

TL;DR. Loro Mirror keeps a typed, immutable app‑state view in sync with a Loro CRDT document. Local setState edits become granular CRDT operations; incoming CRDT events update your state. You keep familiar React patterns and gain collaboration, offline edits, and history.

CRDT: A Conflict‑free Replicated Data Type lets multiple peers edit concurrently and still converge without central coordination.

Local‑first: Data is usable offline and synced later; the device is the primary source of truth.

Overview

Loro is a CRDT library for local‑first apps. It supports rich containers—Text, Map, List/MovableList, MovableTree—with versioning, time‑travel, and compact updates/snapshots.

Though CRDTs ensure CRDTs states converge, apps still need glue code to map between CRDT documents and UI state to ensure their consistency. It's not an easy task.

Loro Mirror addresses this boundary. You declare a schema once. Mirror maintains an immutable app‑state view and handles both directions:

• Event → state. Loro events update your state.
• State → CRDT. setState diffs become container‑level CRDT ops (insert / delete / move / text edits).

For an update, if k items change and each changed item affects m of its immediate fields, time complexity is ≈ O(k·m). (k = number of changed items; m = average number of changed immediate fields per changed item.) This is similar to React’s render complexity.

Why this exists

Without Mirror, projects that uses Loro need to:

1. Map CRDTs states to UI states
2. Diff UI edits and translate them to CRDT operations
3. Subscribe to CRDT events and patch UI state

This code is repetitive and easy to get wrong. Mirror centralizes it behind a declarative schema.

What Mirror provides

• Declarative schema. Describe UI state in terms of Loro containers; Mirror maintains an immutable view.
• Typed and framework‑agnostic. Works in plain TypeScript, React (via loro-mirror-react) or any other UI framework that supports immutable states.
• Fine‑grained diffs. Generates ops such as item moves in MovableList and character deltas in Text.

How to use

1. Define a schema that describes your app state
2. Create a LoroDoc and a Mirror store; provide schema
3. Update via setState. Subscribe for changes if needed.
4. Sync across peers using Loro updates; Mirror applies remote delta back to your app state automatically.

Basic Example

/**
 * As an example, you can use `useState` from React to manage the state
 *
 * `const [appState, setAppState] = useState({});`
 */
function setAppState(state: any) {}
// ---cut---

// 1) Declare state shape – a MovableList of todos with stable Container ID `$cid`
type TodoStatus = "todo" | "inProgress" | "done";
const appSchema = schema({
  todos: schema.LoroMovableList(
    schema.LoroMap({
      text: schema.String(),
      status: schema.String<TodoStatus>(),
    }),
    // $cid is the container ID of LoroMap assigned by Loro
    (t) => t.$cid,
  ),
});

// 2) Create a Loro document and a Mirror store
const doc = new LoroDoc();
const store = new Mirror({
  doc,
  schema: appSchema,
  // InitialState will not be written into LoroDoc
  initialState: { todos: [] },
});

// 3) Subscribe (optional) – know whether updates came from local or remote
const unsubscribe = store.subscribe((state, { direction, tags }) => {
  if (direction === SyncDirection.FROM_LORO) {
    console.log("Remote update", { state, tags });
  } else {
    console.log("Local update", { state, tags });
  }

  // You can use `state` to render directly, it's a new immutable object that shares
  // the unchanged fields with the old state
  setAppState(state);
});

// 4) Either draft‑mutate or return a new state
// Draft‑style (mutate a draft)
store.setState((s) => {
  s.todos.push({ text: "Draft add", status: "todo" });
});

// Immutable return (construct a new object)
store.setState((s) => ({
  ...s,
  todos: [...s.todos, { text: "Immutable add", status: "todo" }],
}));

// 5) Sync across peers with Loro updates (transport‑agnostic)
// Example: two docs in memory – in real apps, send `bytes` over WS/HTTP/WebRTC
const other = new LoroDoc();
other.import(doc.export({ mode: "snapshot" }));

// Wire realtime sync (local updates → remote import)
const stop = doc.subscribeLocalUpdates((bytes) => {
  other.import(bytes);
});

// Any `store.setState(...)` on `doc` now appears in `other` as well

React Example

type TodoStatus = "todo" | "inProgress" | "done";

const todoSchema = schema({
  todos: schema.LoroMovableList(
    schema.LoroMap({
      text: schema.String(),
      status: schema.String<TodoStatus>(),
    }),
    (t) => t.$cid,
  ),
});

export function TodoApp() {
  const doc = useMemo(() => new LoroDoc(), []);
  const { state, setState } = useLoroStore({
    doc,
    schema: todoSchema,
    initialState: { todos: [] },
  });

  function addTodo(text: string) {
    setState((s) => {
      s.todos.push({ text, status: "todo" });
    });
  }

  return (
    <>
      <button onClick={() => addTodo("Write blog")}>Add</button>
      <ul>
        {state.todos.map((t) => (
          <li key={t.$cid}>
            <input
              value={t.text}
              onChange={(e) =>
                setState((s) => {
                  const i = s.todos.findIndex((x) => x.$cid === t.$cid);
                  // Text delta will be calculated automatically
                  if (i !== -1) s.todos[i].text = e.target.value;
                })
              }
            />
            <select
              value={t.status}
              onChange={(e) =>
                setState((s) => {
                  const i = s.todos.findIndex((x) => x.$cid === t.$cid);
                  if (i !== -1)
                    s.todos[i].status = e.target.value as TodoStatus;
                })
              }
            >
              <option value="todo">Todo</option>
              <option value="inProgress">In Progress</option>
              <option value="done">Done</option>
            </select>
          </li>
        ))}
      </ul>
    </>
  );
}

Undo/Redo

// Inside the same component, after creating `doc`:
const undo = useMemo(() => new UndoManager(doc), [doc]);

// Add controls anywhere in your UI:
<div>
  <button onClick={() => undo.undo()}>Undo</button>
  <button onClick={() => undo.redo()}>Redo</button>
  {/* UndoManager only reverts your local edits; remote edits stay. */}
  {/* See docs: <https://loro.dev/docs/advanced/undo> */}
  {/* For full time travel, see: <https://loro.dev/docs/tutorial/time_travel> */}
</div>;

What you get

• Type-safe, framework-agnostic state
• Each mutation becomes a minimal change-set (CRDT delta)—no manual diffing
• Fine-grained updates to subscribers for fast, predictable renders
• Built-in history and time travel
• Offline-first sync via updates or snapshots with deterministic conflict resolution over any transport (HTTP, WebSocket, P2P)
• Collaborative undo/redo across clients

We built a example PWA app here https://todo.loro.dev . It’s open source at https://github.com/loro-dev/loro-todo. It’s collaborative and account-free. The data will be persisted locally in IndexedDB and saved in the cloud for 7 days. You can share your todo list with others by just sharing the unique URL. In the codebase, only a tiny portion of the code is about Loro thanks to the help of loro-mirror.

Where we’re going

Because Mirror owns the bidirectional mapping between application state and the Loro document, we can move value up the stack while lowering integration cost. For example:

• Text. Many interfaces render by lines, yet LoroText’s low‑level API is index‑based. Teams typically re‑implement line segmentation and map edits back to lines by hand. With Mirror in the middle, it becomes feasible to surface optional line‑aware events on top of LoroText so the UI receives stable, line‑based diffs without custom conversion—while retaining the underlying CRDT guarantees.
• Tree. LoroTree CRDT already ensures correct concurrent moves, but developers still translate tree operations into application‑state patches. Mirror carries first‑class mappings from tree events into your state shape, so consumers can work with natural “insert/move/delete node” updates.
• Ephemeral patches. We'll add setStateWithEphemeralPatch so Mirror can stream temporary drag or scale interactions through an EphemeralStore, letting collaborators see live previews while the persisted history stays clean and deduplicated once the change finalizes.

By using loro-mirror to bridge CRDTs and application state consistency, and by expressing schemas declaratively, we can let AI help developers get more done correctly. This makes Loro not only suitable for professional creative tools with real-time collaboration, but also for enabling people to build practical mini-tools for themselves and their communities.

If this work helps you build collaborative, local‑first experiences, we’d be grateful for your sponsorship. You can support us via GitHub Sponsors.

Loro is a Conflict-free Replicated Data Type (CRDT) library that developers can use to implement real-time collaboration and version control in their applications. You can use Loro to create local-first software. Loro 1.0 has a stable data format, excellent performance, and rich features. You can use it in Rust, JS (via WASM), and Swift.

const docA = new LoroDoc();
const docB = new LoroDoc();
docA.setPeerId(0);
docA.setPeerId(1);

docA.getText("text").insert(0, "Hello!");
docB.getText("text").insert(0, "Hi!");
const versionA: Uint8Array = docA.version().encode();
const versionB: Uint8Array = docB.version().encode();

// Exchange versionA and versionB Info
const bytesA: Uint8Array = docA.export({
    mode: "update",
    from: VersionVector.decode(versionB),
});
const bytesB: Uint8Array = docB.export({
    mode: "update",
    from: VersionVector.decode(versionA),
});

// Exchange bytesA and bytesB
docB.import(bytesA);
docA.import(bytesB);

console.log(docA.getText("text").toString()); // Hello!Hi!
console.log(docB.getText("text").toString()); // Hello!Hi!

const docA = new LoroDoc();
const docB = new LoroDoc();
docA.setPeerId(0);
docA.setPeerId(1);
docA.getText("text").insert(0, "Hello!");
docB.getText("text").insert(0, "Hi!");

// Exchange versionA and versionB Info
const bytesA: Uint8Array = docA.export({
    mode: "update",
});
const bytesB: Uint8Array = docB.export({
    mode: "update",
});

// Exchange bytesA and bytesB
docB.import(bytesA);
docA.import(bytesB);

console.log(docA.getText("text").toString()); // Hello!Hi!
console.log(docB.getText("text").toString()); // Hello!Hi!

Features of Loro 1.0

High-performance CRDTs

High-performance, general-purpose CRDTs can significantly reduce data synchronization complexity and are crucial for local-first development.

However, large CRDT documents may face challenges with loading speed and memory consumption, especially when dealing with those with extensive editing histories. Loro 1.0 addresses this challenge through a new storage format, achieving a 10x improvement in loading speed. In benchmarks using Loro with real-world editing data, we've reduced the loading time for a document with millions of operations from 16ms to 1ms. When utilizing the shallow snapshot format (discussed later), the time can be further reduced to 0.37ms. As a result, Loro will not become a bottleneck for applications dealing with such large documents. It expands the potential use cases for CRDTs, making them viable for a wider range of applications.

Rich CRDT types

Loro now supports rich text CRDT, which enhances the merge result of rich text (text with formatting and styling) to better align with user expectations. Our text/list CRDT is based on the Fugue algorithm. It prevents interleaving issues when merging concurrent edits. For example, it can avoid unintended merges like "1H2i3" when "123" and "Hi" are inserted concurrently.

We also support:

• Movable List: Supports set, insert, delete, and move operations. The algorithm ensures that after merging concurrent moves, each element occupies only one position.
• Map: Similar to a JavaScript object.
• Movable Tree: Used to model file directories, outliners, and other hierarchical structures that may need moving. It ensures no cyclic dependencies exist in the tree after merging concurrent move operations.

Loro also supports nesting between types, so you can model edits on JSON documents through them:

You can find all the code samples in this blog here

  LoroDoc,
  LoroList,
  LoroMap,
  LoroText,
} from "npm:loro-crdt@1.0.0-beta.2";

// Create a JSON structure of
interface JsonStructure {
  users: LoroList<
    LoroMap<{
      name: string;
      age: number;
    }>
  >;
  notes: LoroList<LoroText>;
}

const doc = new LoroDoc<JsonStructure>();
const users = doc.getList("users");
const user = users.insertContainer(0, new LoroMap());
user.set("name", "Alice");
user.set("age", 20);
const notes = doc.getList("notes");
const firstNote = notes.insertContainer(0, new LoroText());
firstNote.insert(0, "Hello, world!");

// { users: [ { age: 20, name: "Alice" } ], notes: [ "Hello, world!" ] }
console.log(doc.toJSON());

Version control

Like Git, Loro saves a complete directed acyclic graph (DAG) of edit history. In Loro, the DAG is used to represent the dependencies between edits, similar to how Git represents commit history.

Loro supports primitives that allow users to switch between different versions, fork new branches, edit on new branches, and merge branches.

Based on this operation primitive, applications can build various Git-like capabilities:

• You can merge multiple versions without needing to manually resolve conflicts
• You can rebase/squash updates from the current branch to the target branch (WIP)

const doc = new LoroDoc();
doc.setPeerId("0");
doc.getText("text").insert(0, "Hello, world!");
doc.checkout([{ peer: "0", counter: 1 }]);
console.log(doc.getText("text").toString()); // "He"
doc.checkout([{ peer: "0", counter: 5 }]);
console.log(doc.getText("text").toString()); // "Hello,"
doc.checkoutToLatest();
console.log(doc.getText("text").toString()); // "Hello, world!"

// Simulate a concurrent edit
doc.checkout([{ peer: "0", counter: 5 }]);
doc.setDetachedEditing(true);
doc.setPeerId("1");
doc.getText("text").insert(6, " Alice!");
// ┌───────────────┐     ┌───────────────┐
// │    Hello,     │◀─┬──│     world!    │
// └───────────────┘  │  └───────────────┘
//                    │
//                    │  ┌───────────────┐
//                    └──│     Alice!    │
//                       └───────────────┘
doc.checkoutToLatest();
console.log(doc.getText("text").toString()); // "Hello, world! Alice!"

You can also use doc.fork() to create a separate doc at the current version. It is independent of the current doc, and works like a fork:

const doc = new LoroDoc();
doc.setPeerId("0");
doc.getText("text").insert(0, "Hello, world!");
doc.checkout([{ peer: "0", counter: 5 }]);
const newDoc = doc.fork();
newDoc.setPeerId("1");
newDoc.getText("text").insert(6, " Alice!");
// ┌───────────────┐     ┌───────────────┐
// │    Hello,     │◀─┬──│     world!    │
// └───────────────┘  │  └───────────────┘
//                    │
//                    │  ┌───────────────┐
//                    └──│     Alice!    │
//                       └───────────────┘
doc.import(newDoc.export({ mode: "update" }));
doc.checkoutToLatest();
console.log(doc.getText("text").toString()); // "Hello, world! Alice!"

Leveraging the potential of the Eg-walker

Event Graph Walker (Eg-walker) is a pioneering collaboration algorithm that combines the strengths of Operational Transformation (OT) and CRDT, two widely used algorithms for real-time collaboration.

While OT is centralized and CRDT is decentralized, OT traditionally had an advantage in terms of lower document overhead. CRDTs initially had higher overhead, but recent optimizations have significantly reduced this gap, making CRDTs increasingly competitive. Eg-walker leverages the best aspects of both approaches.

Not only have we use the idea of Eg-walker for Text and List CRDTs in Loro, but Loro's overall architecture has also been greatly inspired by Eg-walker. As a result, Loro closely resembles Eg-walker in terms of algorithmic properties.

The Eg-walker paper was released in September 2023. Prior to its official publication, Joseph Gentle shared an initial version of the algorithm in the Diamond-Type repository. Excited by the design, I implemented a similar algorithm in Loro two years ago. A brief introduction to this algorithm can be found here.

The properties of Eg-walker includes:

• It itself conforms to the definition of CRDT, so it has the strong eventual consistency property of CRDT, thus can be used in distributed environments
• Fast local operation speed: compared to previous CRDTs, it processes operations extremely fast because it doesn't need to generate corresponding Operations based on CRDT data structures
• Fast merging of remote operations: The complexity of OT merging remote operations is O(n^2), while Eg-walker, like mainstream CRDTs, is O(nlogn), only reaching O(n^2) in extremely rare worst-case scenarios. This means that when the number of concurrent operations reaches 10,000, OT will start to show noticeable lag to users, while CRDTs can handle it easily. And in most real-world scenario benchmarks, it's faster than other CRDTs.
• Lower memory usage: Because it doesn't need to persistently store CRDT structures in memory, its memory usage is lower than general CRDTs
• Faster import speed: CRDT documents often take a long time to load because they need to parse the corresponding CRDT structures or operations to build the CRDT data structures. Without these structures, they cannot continue subsequent editing, resulting in long import times. Eg-walker, like OT algorithms, only needs the current document state and does not need to build these additional structures to allow users to start editing the document directly, thus achieving much faster import speed

In the past quarter, we have made significant architectural adjustments to allow Loro to further leverage the advantages of the Eg-walker algorithm. Here are our achievements

Shallow Snapshot

By default, Loro stores the complete editing history of the document like Git, because the Eg-walker algorithm needs to load edits that are parallel to them and to the least common ancestor when merging remote edits. Shallow Snapshot is like Git's Shallow Clone, which can remove old historical operations that users don't need, greatly reducing document size and improving document import and export speed. This allows you to cold store document history that is too old and mainly use shallow doc for collaboration. Here's an example usage:

const doc = new LoroDoc();
for (let i = 0; i < 10_000; i++) {
  doc.getText("text").insert(0, "Hello, world!");
}
const snapshotBytes = doc.export({ mode: "snapshot" });
const shallowSnapshotBytes = doc.export({
  mode: "shallow-snapshot",
  frontiers: doc.frontiers(),
});

console.log(snapshotBytes.length); // 5421
console.log(shallowSnapshotBytes.length); // 869

For details on the implementation principle, see Shallow Snapshot.

Optimized Document Format

Loro version 1.0 has achieved a 10x to 100x improvement in document import speed compared to version 0.16, which already has a fast import speed. It makes it possible to load a large text document with several million operations in under a frame time.

This is because we introduced a new snapshot format. When a LoroDoc is initialized through this snapshot format, we don't parse the corresponding document state and historical information until the user actually needs that information.

In Loro 1.0's snapshot format, without compression algorithms, its document size is twice that of the old version (and other mainstream CRDTs). This additional size mainly comes from encoding historical operations + document state in the 1.0 snapshot format, without reusing stored data between the two, while in the old version we used the order of historical operations to encode the current state of the document (the old version's encoding learned from Automerge encoding's Value Column).

Trading twice the document size for ten times the import speed is worthwhile because import speed affects the performance of many aspects, and the import speed of CRDT documents is often noticeable to users on large documents (> 16ms). It also leaves possibilities for more optimizations in the future.

Inspired by the design of Key-Value Databases, we have also divided the storage of document state and history into blocks, with each block roughly 4KB in size, so that when users really need a piece of history, we only need to decompress and read this 4KB of content, without parsing the entire document. This has led to a qualitative improvement in import speed, and because the serialization format can better compress history and state, memory usage is also lower than before.

The lazy loading optimization takes advantage of Eg-walker's property that "it doesn't need to keep the complete CRDT data structure in memory at all times, and only needs to access historical operations when parallel edits occur".

Benchmarks

All benchmarks results below were performed on a MacBook Pro M1 2020

Below is a comparison of Snapshot import and export speeds between Loro versions 1.0.0-beta.1 and 0.16.12. The benchmark is based on document editing history from the real world. Thanks to latch.bio for sharing the document data. The benchmark code is available here. The document contains 1,659,541 operations.

In Loro, a Snapshot stores the document history along with its current state. The Shallow Snapshot format, similar to Git's Shallow Clone, can exclude history. In the benchmark below, the Shallow Snapshot has a depth=1 (only the most recent operation history is retained, other historical operations are removed)

Here are the key points of this benchmark:

• The Shallow Snapshot has a depth of 1, meaning it only contains the document state and a single historical operation, which is why it's significantly faster
• GetAllValue refers to calling doc.get_deep_value() (in JS, it's doc.toJSON() ). It loads the complete state of the document and obtains the corresponding JSON-like structure. This represents the time spent on CRDT parsing before a user loads a document.
• Edit refers to making a local modification. As you can see, it has little impact on the time taken because Loro doesn't need to load the complete CRDT data structure for local operations.
• Export refers to exporting the complete document data again. We expect to further reduce the time spent here in the future, as we can continue to reuse the encoding of unmodified Blocks from the import.

The following shows the performance on a document after applying the editing history from the Automerge Paper 100 times. You can reproduce the results here. The document contains:

• 18,231,500 single-character insertion operations
• 7,746,300 single-character deletion operations
• 25,977,800 operations totally
• 10,485,200 characters in the final document

• The New snapshot data is smaller because it performs additional simple compression on each Block during encoding internally

Next Steps for Loro

Loro Version Controller

Importing Loro Git Repo into Loro Version Controller

Loro's performance on a single document is now sufficient to cover the real-time collaboration and version management needs of most documents. So our next step will be to explore real-time collaboration and version control across a collection of documents.

We believe that CRDTs can create a Git for Everyone and Everything:

• It's for Everyone because by leveraging the power of CRDTs, we can make version control much easier to reason about and use for the average person.
• It's (nearly) for Everything because Loro provides a rich set of data synchronization types. We're no longer limited to synchronizing plain text data, but can solve semantic automatic merging of JSON-like schema, which can meet most needs of creative tools and collaborative tools.

We've created a demo of the Loro version controller, which is based on our sub-document implementation (implemented in the application layer) with Version information. It can import the entire React repository (about 20,000 commits, thousands of collaborators), and it supports real-time collaboration on such repositories. However, how to better manage versions and seamlessly integrate with Git still needs to be explored.

Loro CRDTs still have significant room for optimization in these scenarios. Currently, the Loro CRDTs library doesn't involve network or disk I/O, which enhances its ease of use but also constrains its capabilities and potential optimizations. For example, while we've implemented block-level storage, documents are still imported and exported as whole units. Adding I/O capabilities to selectively load/save blocks would enable significant performance optimizations.

Conclusion

Loro 1.0 features great performance improvements, rich CRDT types, and advanced version control features. Our optimized document format has yielded promising results on the import speed and the memory usage.

Now that Loro CRDTs are stable, we are able to develop a better ecosystem. We're excited to see it being applied in various scenarios. If you're interested in using Loro, welcome to join our Discord community for discussions.

This article introduces the implementation difficulties and challenges of Movable Tree CRDTs when collaboration, and how Loro implements it and sorts child nodes. The algorithm has high performance and can be used in production.

Background

In distributed systems and collaborative software, managing hierarchical relationships is difficult and complex. Challenges arise in resolving conflicts and meeting user expectations when working with the data structure that models movement by combining deletion and insertion. For instance, if a node is concurrently moved to different parents in replicas, it may lead to the unintended creation of duplicate nodes with the same content. Because the node is deleted twice and created under two parents.

Currently, many software solutions offer different levels of support and functionality for managing hierarchical data structures in distributed environments. The key variation among these solutions lies in their approaches to handling potential conflicts.

Conflicts in Movable Trees

A movable tree has 3 primary operations: creation, deletion, and movement. Consider a scenario where two peers independently execute various operations on their respective replicas of the same movable tree. Synchronizing these operations can lead to potential conflicts, such as:

• The same node was deleted and moved
• The same node was moved under different nodes
• Different nodes were moved, resulting in a cycle
• The ancestor node is deleted while the descendant node is moved

Deletion and Movement of the Same Node

This situation is relatively easy to resolve. It can be addressed by applying one of the operations while ignoring the other based on the timestamp in the distributed system or the application's specific requirements. Either approach yields an acceptable outcome.

Moving the Same Node Under Different Parents

Merging concurrent movement operations of the same node is slightly more complex. Different approaches can be adopted depending on the application:

• Delete the node and create copies of nodes under different parent nodes. Subsequent operations then treat these nodes independently. This approach is acceptable when node uniqueness is not critical.
• Allow the node have two edges pointing to different parents. However, this approach breaks the fundamental tree structure and is generally not considered acceptable.
• Sort all operations, then apply them one by one. The order can be determined by timestamps in a distributed system. Providing the system maintains a consistent operation sequence, it ensures uniform results across all peers.

Movement of Different Nodes Resulting in a Cycle

Concurrent movement operations that cause cycles make the conflict resolution of movable trees complex. Matthew Weidner listed several solutions to resolve cycles in his blog.

1. Error. Some desktop file sync apps do this in practice (Martin Kleppmann et al. (2022) give an example).
2. Render the cycle nodes (and their descendants) in a special “time-out” zone. They will stay there until some user manually fixes the cycle.
3. Use a server to process move ops. When the server receives an op, if it would create a cycle in the server’s own state, the server rejects it and tells users to do likewise. This is what Figma does. Users can still process move ops optimistically, but they are tentative until confirmed by the server. (Optimistic updates can cause temporary cycles for users; in that case, Figma uses strategy (2): it hides the cycle nodes.)
4. Similar, but use a topological sort (below) instead of a server’s receipt order. When processing ops in the sort order, if an op would create a cycle, skip it (Martin Kleppmann et al. 2022).
5. For forests: Within each cycle, let B.parent = A be the edge whose set operation has the largest LWW timestamp. At render time, “hide” that edge, instead rendering B.parent = "none", but don’t change the actual CRDT state. This hides one of the concurrent edges that created the cycle. • To prevent future surprises, users’ apps should follow the rule: before performing any operation that would create or destroy a cycle involving a hidden edge, first “affirm” that hidden edge, by performing an op that sets B.parent = "none".
6. For trees: Similar, except instead of rendering B.parent = "none", render the previous parent for B - as if the bad operation never happened. More generally, you might have to backtrack several operations. Both Hall et al. (2018) and Nair et al. (2022) describe strategies along these lines.

Ancestor Node Deletion and Descendant Node Movement

The most easily overlooked scenario is moving descendant nodes when deleting an ancestor node. If all descendant nodes of the ancestor are deleted directly, users may easily misunderstand that their data has been lost.

How Popular Applications Handle Conflicts

Dropbox is a file data synchronization software. Initially, Dropbox treated file movement as a two-step process: deletion from the original location followed by creation at a new location. However, this method risked data loss, especially if a power outage or system crash occurred between the delete and create operations.

Today, when multiple people move the same file concurrently and attempt to save their changes, Dropbox detects a conflict. In this scenario, it typically saves one version of the original file and creates a new "conflicted copy" for the changes made by one of the users.

The image shows the conflict that occurs when A is moved to the B folder and B is moved to the A folder concurrently.

Figma is a real-time collaborative prototyping tool. They consider tree structures as the most complex part of the collaborative system, as detailed in their blog post about multiplayer technology. To maintain consistency, each element in Figma has a "parent" attribute. The centralized server plays a crucial role in ensuring the integrity of these structures. It monitors updates from various users and checks if any operation would result in a cycle. If a potential cycle is detected, the server rejects the operation.

However, due to network delays and similar issues, there can be instances where updates from users temporarily create a cycle before the server has the chance to reject them. Figma acknowledges that this situation is uncommon. Their solution is straightforward yet effective: they temporarily preserve this state and hide the elements involved in the cycle. This approach lasts until the server formally rejects the operation, ensuring both the stability of the system and a seamless user experience.

An animation that demonstrates how Figma resolves conflicts.

Movable Tree CRDTs

The applications mentioned above use movable trees and resolve conflicts based on centralized solutions. Another alternative approach to collaborative tree structures is using Conflict-free Replicated Data Types (CRDTs). While initial CRDT-based algorithms were challenging to implement and incurred significant storage overhead as noted in prior research, such as Abstract unordered and ordered trees CRDT or File system on CRDT, but continual optimization and improvement have made several CRDT-based tree synchronization algorithms suitable for certain production environments. This article highlights two innovative CRDT-based approaches for movable trees. The first is presented by Martin Kleppmann et al. in their work A highly-available move operation for replicated trees and the second by Evan Wallace in his CRDT: Mutable Tree Hierarchy.

A highly-available move operation for replicated trees

This paper unifies the three operations used in trees (creating, deleting, and moving nodes) into a move operation. The move operation is defined as a four-tuple Move t p m c, where t is the operation's unique and ordered timestamp such as Lamport timestamp, p is the parent node ID, m is the metadata associated with the node, and c is the child node ID.

If all nodes of the tree do not contain c, this is a creation operation that creates a child node c under parent node p. Otherwise, it is a move operation that moves c from its original parent to the new parent p. Additionally, node deletion is elegantly handled by introducing a designated TRASH node; moving a node to TRASH implies its deletion, with all descendants of TRASH considered deleted. But they remain in memory to prevent concurrent editing from moving them to other nodes. In order to handle the previously mentioned situation of deleting ancestor nodes and moving descendant nodes concurrently.

In the three potential conflicts mentioned earlier, since deletion is also defined as a move operation, deleting and moving the same node is transformed into two move operations, leaving only two remaining problems:

• Moving the same node under different parents
• Moving different nodes, creating a cycle

Logical timestamps are added so that all operations can be linearly ordered, thus the first conflict can be avoided as they can be expressed as two operations in sequence rather than concurrently for the same node. Therefore, in modeling a Tree using only move operations, the only exceptional case in concurrent editing would be creating a cycle, and operations causing a cycle are termed unsafe operations.

This algorithm sorts all move operations according to their timestamps. It can then sequentially apply each operation. Before applying, the algorithm detects cycles to determine whether an operation is safe. If the operation creates a cycle, we ignore the unsafe operation to ensure the correct structure of the tree.

Based on the above approach, the consistency problem of movable trees becomes the following two questions:

1. How to introduce global order to operations
2. How to apply a remote operation that should be inserted in the middle of an existing sorted sequence of operations

Globally Ordered Logical Timestamps

Lamport Timestamp can determine the causal order of events in a distributed system. Here's how they work: each peer starts with a counter initialized to 0. When a local event occurs, the counter is increased by 1, and this value becomes the event's Lamport Timestamp. When peer A sends a message to peer B, A attaches its Lamport Timestamp to the message. Upon receiving the message, peer B compares its current logical clock value with the timestamp in the message and updates its logical clock to the larger value.

To globally sort events, we first look at the Lamport Timestamps: smaller numbers mean earlier events. If two events have the same timestamp, we use the unique ID of the peer serves as a tiebreaker.

Apply a Remote Operation

An op's safety depends on the tree's state when applied, avoiding cycles. Insertion requires evaluating the state formed by all preceding ops. For remote updates, we may need to:

1. Undo recent ops
2. Insert the new op
3. Reapply undone ops

This ensures proper integration of new ops into the existing sequence.

Undo Recent Ops

Since we've modeled all operations on the tree as move operations, undoing a move operation involves either moving the node back to its old parent or undoing the operation that created this node. To enable quick undoing, we cache and record the old parent of the node before applying each move operation.

Apply the Remote Op

Upon encountering an unsafe operation, disregarding its effects prevents the creation of a cycle. Nevertheless, it's essential to record the operation, as the safety of an operation is determined dynamically. For instance, if we receive and sort an update that deletes another node causing the cycle prior to this operation, the operation that was initially unsafe becomes safe. Additionally, we need to mark this unsafe operation as ineffective, since during undo operations, it's necessary to query the old parent node, which is the target parent of the last effective operation in the sequence targeting this node.

Reapply Undone Ops

Cycles only occur when receiving updates from other peers, so the undo-do-redo process is also needed at this time. When receiving a new op:

function apply(newOp)
      // Compare the ID of the new operation with existing operations
      if largerThanExistingOpId(newOp.id, oplog)
          // If the new operation's ID is greater, apply it directly
          oplog.applyOp(newOp)
      else
          // If the new operation's ID is not the greatest, undo operations until it can be applied
          undoneOps = oplog.undoUtilCanBeApplied(newOp)
          oplog.applyOp(newOp)
          // After applying the new operation, redo the undone operations to maintain sequence order
          oplog.redoOps(undoneOps)

• If the new operation depends on an op that has not been encountered locally, indicating that some inter-version updates are still missing, it is necessary to temporarily cache the new op and wait to apply it until the missing updates are received.
• Compare the new operation with all existing operations. If the opId of the new operation is greater than that of all existing operations, it can be directly applied. If the new operation is safe, record the parent node of the target node as the old parent node, then apply the move operation to change the current state. If it is not safe, mark this operation as ineffective and ignore the operation's impact.
• If the new opId is sorted in the middle of the existing sequence, it is necessary to pop the operations that are sorted later from the sequence one by one, and undo the impact of this operation, which means moving back to the child of the old parent node, until the new operation can be applied. After applying the new operation, reapply the undone nodes in sequence order, ensuring that all operations are applied in order.

The following animated GIF demonstrates the process executed by Peer1:

1. Received Peer0 creating node A with the root node as its parent.
2. Received Peer0 creating node B with A as its parent.
3. Created node C with A as its parent and synchronized it with Peer0.
4. Moved C to have B as its parent.
5. Received Peer0's moving B to have C as its parent.

The queue at the top right of the animation represents the order of local operations and newly received updates. The interpretation of each element in each Block is as follows:

A particular part of this process to note is the two operations with lamport timestamps of 0:3 and 1:3. Initially, the 1:3 operation moving C to B was created and applied locally, followed by receiving Peer0's 0:3 operation moving B to C. In lamport timestamp order, 0:3 is less than 1:3 but greater than 1:2 (with peer as the tiebreaker when counters are equal). To apply the new op, the 1:3 operation is undone first, moving C back to its old parent A, then 0:3 moving B to C is applied. After that, 1:3 is redone, attempting to move C to B again (the old parent remains A, omitted in the animation). However, a cycle is detected during this attempt, preventing the operation from taking effect, and the state of the tree remains unchanged. This completes an undo-do-redo process.

CRDT: Mutable Tree Hierarchy

Evan Wallace has developed an innovative algorithm that enables each node to track all its historical parent nodes, attaching a counter to each recorded parent. The count value of a new parent node is 1 higher than that of all the node's historical parents, indicating the update sequence of the node's parents. The parent with the highest count is considered the current parent node.

During synchronization, this parent node information is also synced. If a cycle occurs, a heuristic algorithm reattaches the nodes causing the cycle back to the nearest historical parent node that won't cause a cycle and is connected to the root node, thus updating the parent node record. This process is repeated until all nodes causing cycles are reattached to the tree, achieving all replica synchronization of the tree structure. The demo in Evan's blog clearly illustrates this process.

As Evan summarized at the end of the article, this algorithm does not require the expensive undo-do-redo process. However, each time a remote move is received, the algorithm needs to determine if all nodes are connected to the root node and reattach the nodes causing cycles back to the tree, which can perform poorly when there are too many nodes.

I established a benchmark to compare the performance of the movable tree algorithms.

Movable Tree CRDTs implementation in Loro

Loro implements the algorithm proposed by Martin Kleppmann et al., A highly-available move operation for replicated trees. On one hand, this algorithm has high performance in most real world scenarios. On the other hand, the core undo-do-redo process of the algorithm is highly similar to how Eg-walker (Event Graph Walker) applies remote updates in Loro. Introduction about Eg-walker can be found in our previous blog.

Movable tree has been introduced in detail, but there is still another problem of tree structure that has not been solved. For movable tree, in some real use cases, we still need the capability to sort child nodes. This is necessary for outline notes or layer management in graphic design softwares. Users need to adjust node order and sync it to other collaborators or devices.

We integrated the Fractional Index algorithm into Loro and combined it with the movable tree, making the child nodes of the movable tree sortable.

There are many introductions to Fractional Index on the web, You can read more about Fractional Index in the Figma blog or Evan blog. In simple terms, Fractional Index assigns a sortable value to each object, and if a new insertion occurs between two objects, the Fractional Index of the new object will be between the left and right values. What we want to speak about more here is how to deal with potential conflicts brought by Fractional Index in CRDTs systems.

Potential Conflicts in Child Node Sorting

As our applications are in a distributive condition, when multiple peers insert new nodes in the same position, the same Fractional Index would be assigned to these differing content but same position nodes. When updates from the remote are applied to local, conflicts arise as the same Fractional Index is encountered.

In Loro, we retain these identical Fractional Index and use PeerID (unique ID of every Peer) as the tie-breaker for the relative order judgment of the same Fractional Index.

Although this solved the sorting problem among the same Fractional Index nodes from different peers, it impacted the generation of new Fractional Index as we cannot generate a new Fractional Index between two same ones. We use two methods to solve this problem:

1. The first method, as stated in Evan's blog, we could add a certain amount of jitter to each generated Fractional Index, (for the ease of explanation, all examples below take decimal fraction as the Fractional Index) for example, when generating a new Fractional Index between 0 and 1, it should have been 0.5, but through random jitters, it could be 0.52712, 0.58312, 0.52834, etc., thus significantly reducing the chance of same Fractional Index appearing.
2. If the situation arises where the same Fractional Index is present on both sides, we can handle this problem by resetting these Fractional Index. For example, if we need to insert a new node between 0.7@A and 0.7@B (which indicates Fractional Index @ PeerID), instead of generating a new Fractional Index between 0.7 and 0.7, we could assign two new Fractional Index respectively for the new node and the 0.7@B node between 0.7 and 1, which could be understood as an extra move operations.

Implementation and Encoding Size

Introducing Fractional Index brings the advantage of node sequence. What about encoding size?

Loro uses drifting-in-space Fractional Index implementation based on Vec<u8>, which is base 256. In other words, you need to continuously insert 128 values forward or backward from the default value to increase the byte size of the Fractional Index by 1. The worst storage overhead case, such as inserting new values alternately each time. For example, the initial sequence is ab, insert c between a and b, then insert d between c and b, then e between c and d, like:

ab    // [128] [129, 128]
acb   // [128] [129, 127, 128] [129, 128]
acdb  // [128] [129, 127, 128] [129, 127, 129, 128] [129, 128]
acedb // [128] [129, 127, 128] [129, 127, 129, 127, 128] [129, 127, 129, 128] [129, 128]

a new operation would cause an additional byte to be needed. But such a situation is very rare.

Considering that potential conflicts wouldn't appear frequently in most applications, Loro simply extended the implementation, the original implementation produced new Fractional Index in Vec<u8> by only increasing or decreasing 1 in certain index to achieve relative sorting. The simple jitter solution was added, by appending random bytes in length of jitter value to Fractional Index. To enable jitter in js, you can use doc.setFractionalIndexJitter(number) with a positive value. But this will increase the encoding size slightly, but each Fractional Index only adds jitter bytes. If you want to generate Fractional Index at the same position with 99% probability without conflict, the relationship between jitter settings and the maximum number of concurrent edits n will be:

When there are numerous Fractional Indexes, there will be many common prefixes after being sorted, when Loro encodes these Fractional Indexes, prefix optimization would be implemented. Each Fractional Index only saves the amount of same prefix bits and remaining bytes with the previous one, which further downsizes the overall encoding size.

Related work

Other than using Fractional Index, there are other movable list CRDT that can make sibling nodes of the tree in order. One of these algorithms is Martin Kleppmann's Moving Elements in List CRDTs, which has been used in Loro's Movable List.

In comparison, the implementation of Fractional Index solution is simpler, and no stable position representation is provided for child nodes when modeling nodes in a tree, otherwise, the overall tree structure would be too complex. However, the Fractional Index has the problem of interleaving, but this is acceptable when some only need relative order and do not require strict sequential semantics, such as figma layer items, multi-level bookmarks, etc.

Benchmark

We conducted performance benchmarks on the Movable Tree implementation by Loro, including scenarios of random node movement, switching to historical versions, and performance under extreme conditions with significantly deep tree structures. The results indicate that it is capable of supporting real-time collaboration and enabling seamless historical version checkouts.

Test environment: M2 Max CPU, you can find the bench code here.

Usage

let doc = new Loro();
let tree: LoroTree = doc.getTree("tree");
let root: LoroTreeNode = tree.createNode();
// By default, append to the end of the parent node's children list
let node = root.createNode();
// Specify the child's position
let node2 = root.createNode(0);
// Move `node2` to be the last child of `node`
node2.move(node);
// Move `node` to be the first child of `node2`
node.move(node2, 0);
// Move the node to become the root node
node.move();
// Move the node to be positioned after another node
node.moveAfter(node2);
// Move the node to be positioned before another node
node.moveBefore(node2);
// Retrieve the index of the node within its parent's children
let index = node.index();
// Get the `Fractional Index` of the node
let fractionalIndex = node.fractionalIndex();
// Access the associated data map container
let nodeData: LoroMap = node.data;

Demo

We developed a simulated Todo app with data synchronization among multiple peers using Loro, including the use of Movable Tree to represent subtask relationships, Map to represent various attributes of tasks, and Text to represent task titles, etc. In addition to basic creation, moving, modification, and deletion, we also implemented version switching based on Loro. You can drag the scrollbar to switch between all the historical versions that have been operated on.