Internals: history & transfer

History is what Tomo is really about. This page covers how it stores versions cheaply, keeps capture off the latency path, and moves big files without blocking small edits.

The content-addressed store #

Every version goes into a content-addressed store: the file is split into content-defined chunks with FastCDC, each chunk is identified by its BLAKE3 hash, compressed with zstd, and both the chunk bytes and the version metadata (vector clocks, conflict records, the path index) live in a single SQLite database under .tomo/db/.

file → FastCDC chunks → BLAKE3 ids → zstd → SQLite, with content-addressed dedup

Why an edit costs ~1% #

FastCDC cuts chunk boundaries based on content, not fixed offsets, so inserting or deleting bytes shifts only the chunks around the edit — not every boundary after it (the flaw that sinks fixed-size chunking). With Tomo's parameters — 16 KiB minimum, 64 KiB average, 256 KiB maximum — a 10 MiB file is roughly 160 chunks. A one-character edit rewrites a single chunk; the other ~159 hash to ids the store already holds and are skipped. New bytes stored: about one 64 KiB chunk, well under 1% of the file. The store even tracks the exact count of newly written chunks as a dedup metric, and the same content hash makes an exec-bit-only change (a chmod) store zero new content — only the mode column changes.

Adaptive capture #

How often should a save become a version? Every keystroke is too much; once a minute loses work. Tomo treats it like a congestion controller. Every canonical change enters a per-file staging buffer, and a flush interval decides when it becomes a version. Under light load that interval is a tiny window; under a storm of saves it escalates, coalescing a burst into one checkpoint; when things go quiet it decays back down.

the capture ladder: escalate under storms, decay back to ~immediate when idle

Two guarantees are load-bearing, and both are property-tested against synthetic storms:

• Live sync latency is untouched. Debouncing applies to history capture only — the sync path always ships the newest bytes right away.
• The final state of every burst is always versioned. Coalescing may drop noisy intermediates (a truncate-then-write's 0-byte flicker, vim's 4913 probe), never the last state. Staleness is bounded by the current rung.

Rung 0's window is a small min_capture_window_ms (default 75 ms) rather than a hard zero, so every real save still becomes a version — it just flushes ~75 ms later, letting a same-path truncate+write pair collapse into its single final state. (In every-change mode the window is a literal 0 ms.)

Chunked, interleaved transfer #

Small edits and giant files share one connection, and a big transfer must never block a quick one behind it. The threshold is 1 MiB:

• A change under 1 MiB rides inline in a single Change frame.
• At or above 1 MiB the sender ships a ChangeManifest — the change plus the ordered list of FastCDC chunk hashes (the same 16/64/256 KiB params and BLAKE3 ids as the store, so the manifest is CAS-coherent). The receiver requests only the chunks it lacks, in batches of 32, and the sender answers a few at a time — so a live small-file Change always interleaves between chunk batches instead of waiting behind the whole file.

The dedup is end-to-end: a chunk the peer already has is never resent. Measured on this design: small-file latency stayed under 7 seconds during a sustained 1 GiB transfer under churn (acceptance scenario 11).

Crash safety #

A kill -9 at any instant must never leave a torn file at a real path or a corrupt history DB. Tomo guarantees this structurally:

• Staging + atomic rename. Every write lands first under .tomo/staging/ and is atomically renamed into place. A crash mid-write leaves only garbage in staging — never a partial file at its final path.
• Absorb only on completion. A chunked transfer is staged under .tomo/staging/chunks/ and absorbed into the engine only once every chunk verifies and the reassembled whole-file hash matches the signature. Absorbing early would persist an index state the disk doesn't have — so a crash mid-assembly can't make the restart scan read a phantom deletion.
• Degrade, don't die. An inbound apply that hits ENOSPC (a full disk) abandons its partial staging and stalls loudly instead of crashing; when space frees up, the peer's reconcile re-ships the uncovered head and it self-heals. Proven end-to-end on a 24 MiB loopback filesystem (scenario 21).

You can check the store's integrity any time with tomo db check, which re-hashes every recorded chunk against its BLAKE3 id.