Architecture

How a mount-rs server stays fast.

mount-rs serve-http serves drives over HTTP. Each drive is owned by one server at a time, which keeps the drive’s namespace in memory and commits changes to shared metadata and block storage. Any server can accept a request and forward it, in one hop, to the owner.

01 / a cluster

Forwarding: any node, one hop to the owner.

Three nodes sit behind one load balancer. The load balancer does not know which server owns a drive: owners publish their drives in Redis, ingresses cache those routes, and every node holds open, mutually authenticated HTTP/3 connections to the others. Send a request and follow it, then stop a node and watch a standby take over its drive.

Clustering is built into serve-http. A cluster needs three things: a membership file that pins every server’s key for mutual TLS, which servers re-read on SIGHUP or a timer, so nodes join and leave without a restart; a Redis route registry, reached over TLS with a password (plaintext is refused except to the same host unless you allow it); and metadata and block storage that every candidate owner of a drive can reach.

In return, any node accepts any request, and with failover turned on a drive whose owner stops moves to a standby with no operator involved. Measured with real binaries, SQLite metadata and a 20-second lease, the new owner was published about 21 seconds after kill -9, and within a second of a clean stop. With the default 240-second lease a crash takes about four minutes, because a standby cannot take a drive before the old lease expires.

Limits: failover is opt-in and needs split-store drives; sessions and requests in flight on a failed owner are lost, not replayed; the registry can be a Redis Sentinel deployment but not Redis Cluster; and servers of different releases cannot share a cluster, because the server-to-server protocol is internal, so a cluster is upgraded by stopping every server. See the failover design and the cluster configuration.

Three mount-rs nodes behind one load balancerClients reach a load balancer that sends each request to any running node. Every node is connected to the other two over HTTP/3, and all three share a Redis route registry, a metadata store and an object store. Each drive is owned by one node at a time; when that node stops, a standby takes the drive over. The controls and the step list beside the diagram give the same information as text.HTTP/3 peer mesh · mutual TLS · pooledClientsone address, a token per driveLoad balancerany healthy node · no drive tableNode 1serve-httpdrive-aowner · namespace in memoryserved 0 · forwarded 0Node 2serve-httpdrive-bowner · namespace in memoryserved 0 · forwarded 0Node 3serve-httpdrive-cowner · namespace in memoryserved 0 · forwarded 0Redis route registrywhich node owns each driveMetadata storeTiDB here: namespaces, leasesObject storeRustFS here: immutable blocks
client traffic node-to-node HTTP/3 shared servicesSwipe the diagram sideways to see all three nodes.Here every node is a failover candidate for every drive. Stop a node, with its button or by clicking it, and a standby takes its drives once it holds the storage writer lease. A restarted node comes back as a standby: there is no automatic fail-back by default.
Drive
Operation
Load balancer sends it to
Route cache on the receiving node
Failover: stop a node

Pick a drive and an operation, then send a request. Choose a node that does not own the drive to see forwarding, or run traffic to watch the load balancer share requests across all three. Stop a node to watch a standby take over its drive, and send a request while the takeover is running to see the retryable 503.

  1. Any node accepts the request. The client sends its drive token over HTTPS to whichever node the load balancer picks.
  2. The route is usually cached. On a miss, the ingress reads the drive’s route from Redis; concurrent misses for one drive share a single lookup. Active routes are re-confirmed in the background before they expire.
  3. One HTTP/3 hop. The request is streamed over one of the ingress’s pooled QUIC connections to the owner, authenticated in both directions with pinned keys from the membership file.
  4. The owner decides. It re-checks the original token, that the route still names it, and its current fencing generation, then streams the response back through the ingress.

A forwarded write whose outcome is unknown, because the connection broke after the owner may have received it, is reported as an error and never replayed. Stopping an owner withdraws its routes, so its drives answer 503 at once instead of after a connection timeout. While no owner is published the 503 carries Retry-After: 1 and means nothing was executed. Without failover the drive stays unavailable until its owner is restarted. With it, a standby that reads no route opens the drive, which succeeds only once storage grants the writer lease, and then publishes a route with a higher fence. Redis decides nothing about who may write.

02 / one node

A request, from the socket to storage.

The listener, token check and routing decision are the same on every node. What differs is whether this node owns the drive. Select any stage to see what it does and what it costs.

Every request

Yes: serve it here

No: forward it once

Own this drive?

The one decision that differs between nodes

Every node runs the same code. The only question is whether this process holds the drive’s fenced writer lease. If it does, the request never leaves the process unless it changes something.

  • Exactly one owner per drive
  • Assigned by the operator, or taken over by a failover candidate
  • Owners publish their drives as routes

03 / caching

Seven layers between a request and the object store.

Most requests are answered in the top three layers. Pick a request to see how far down it has to go, and open a layer to see what a hit there saves.

Follow a request

The ingress finds the owner in its route cache, and the owner answers from memory. Nothing below layer 3 is touched. Measured p99: 0.72 ms.

04 / writes

Durable by default, writeback by choice.

Blocks always reach storage before metadata refers to them, and every metadata commit carries the writer’s fence. Writeback moves the acknowledgement earlier; it does not change that order.

Write mode

The response means the change is committed.

Request. A PUT, mkdir, rename or delete reaches the drive’s owner.

Both paths fail closed: if the lease or fence is lost, the owner stops accepting operations and a replacement reloads authoritative metadata. An uncertain commit is reported as an error and never replayed.

What makes it fast

Remove round trips from the common path.

None of these needs a special client. They come from giving every drive a single owner and keeping each layer’s job narrow.

01

Metadata reads stay in memory

Each drive has exactly one owner holding a fenced writer lease. While the lease holds, the owner’s in-memory namespace is authoritative, so stat, head and list never wait on the database.

02

Any node, one hop

A request can land on any node. If that node does not own the drive it forwards once, over pooled HTTP/3 connections to the owner, and streams the response back. There is no second hop and no redirect to the client.

03

Routes are already known

Ingresses cache which node owns each drive and confirm active routes in the background, so forwarded requests skip the Redis lookup. On the cloud benchmark this lowered forwarded read p99 from 0.87 ms to 0.72 ms.

04

Leases renew off the request path

An owner renews its 240-second lease every 30 seconds in the background and serves within a 60-second window, instead of paying a lease round trip per operation. The same lease is what lets a standby take the drive over if the owner stops.

05

One statement per durable write

A durable mutation finishes its blocks, then commits the new revision with a single conditional update that checks both the revision and the writer fence. Nothing is locked across drives.

06

Small probes, not big reads

Checking whether a TiDB namespace changed reads a covering index entry, not the whole namespace row. That cut TiKV traffic per unchanged read from 1.74 MB to 4.76 KB.

07

Immutable, cacheable blocks

File bytes live in immutable blocks, cached in bounded RAM and disk tiers in front of the object store. Uploads stream their blocks to storage as they arrive, up to 5 GiB per file.

08

Writeback when you want it

Drives that can accept an acknowledgement before durability can use writeback: mutations are applied in memory and published later, with p99 under 1 ms even when forwarded.