Metadata reads stay in memory
Each drive has exactly one owner holding a fenced writer lease. While the lease holds, the owner’s in-memory namespace is authoritative, so stat, head and list never wait on the database.
Architecture
mount-rs serve-http serves drives over HTTP. Each drive is owned by one server at a time, which keeps the drive’s namespace in memory and commits changes to shared metadata and block storage. Any server can accept a request and forward it, in one hop, to the owner.
01 / a cluster
Three nodes sit behind one load balancer. The load balancer does not know which server owns a drive: owners publish their drives in Redis, ingresses cache those routes, and every node holds open, mutually authenticated HTTP/3 connections to the others. Send a request and follow it, then stop a node and watch a standby take over its drive.
Clustering is built into serve-http. A cluster needs three things: a membership file that pins every server’s key for mutual TLS, which servers re-read on SIGHUP or a timer, so nodes join and leave without a restart; a Redis route registry, reached over TLS with a password (plaintext is refused except to the same host unless you allow it); and metadata and block storage that every candidate owner of a drive can reach.
In return, any node accepts any request, and with failover turned on a drive whose owner stops moves to a standby with no operator involved. Measured with real binaries, SQLite metadata and a 20-second lease, the new owner was published about 21 seconds after kill -9, and within a second of a clean stop. With the default 240-second lease a crash takes about four minutes, because a standby cannot take a drive before the old lease expires.
Limits: failover is opt-in and needs split-store drives; sessions and requests in flight on a failed owner are lost, not replayed; the registry can be a Redis Sentinel deployment but not Redis Cluster; and servers of different releases cannot share a cluster, because the server-to-server protocol is internal, so a cluster is upgraded by stopping every server. See the failover design and the cluster configuration.
Pick a drive and an operation, then send a request. Choose a node that does not own the drive to see forwarding, or run traffic to watch the load balancer share requests across all three. Stop a node to watch a standby take over its drive, and send a request while the takeover is running to see the retryable 503.
A forwarded write whose outcome is unknown, because the connection broke after the owner may have received it, is reported as an error and never replayed. Stopping an owner withdraws its routes, so its drives answer 503 at once instead of after a connection timeout. While no owner is published the 503 carries Retry-After: 1 and means nothing was executed. Without failover the drive stays unavailable until its owner is restarted. With it, a standby that reads no route opens the drive, which succeeds only once storage grants the writer lease, and then publishes a route with a higher fence. Redis decides nothing about who may write.
02 / one node
The listener, token check and routing decision are the same on every node. What differs is whether this node owns the drive. Select any stage to see what it does and what it costs.
Every request
Yes: serve it here
No: forward it once
Own this drive?
Every node runs the same code. The only question is whether this process holds the drive’s fenced writer lease. If it does, the request never leaves the process unless it changes something.
03 / caching
Most requests are answered in the top three layers. Pick a request to see how far down it has to go, and open a layer to see what a hit there saves.
The ingress finds the owner in its route cache, and the owner answers from memory. Nothing below layer 3 is touched. Measured p99: 0.72 ms.
04 / writes
Blocks always reach storage before metadata refers to them, and every metadata commit carries the writer’s fence. Writeback moves the acknowledgement earlier; it does not change that order.
The response means the change is committed.
Request. A PUT, mkdir, rename or delete reaches the drive’s owner.
Both paths fail closed: if the lease or fence is lost, the owner stops accepting operations and a replacement reloads authoritative metadata. An uncertain commit is reported as an error and never replayed.
What makes it fast
None of these needs a special client. They come from giving every drive a single owner and keeping each layer’s job narrow.
Each drive has exactly one owner holding a fenced writer lease. While the lease holds, the owner’s in-memory namespace is authoritative, so stat, head and list never wait on the database.
A request can land on any node. If that node does not own the drive it forwards once, over pooled HTTP/3 connections to the owner, and streams the response back. There is no second hop and no redirect to the client.
Ingresses cache which node owns each drive and confirm active routes in the background, so forwarded requests skip the Redis lookup. On the cloud benchmark this lowered forwarded read p99 from 0.87 ms to 0.72 ms.
An owner renews its 240-second lease every 30 seconds in the background and serves within a 60-second window, instead of paying a lease round trip per operation. The same lease is what lets a standby take the drive over if the owner stops.
A durable mutation finishes its blocks, then commits the new revision with a single conditional update that checks both the revision and the writer fence. Nothing is locked across drives.
Checking whether a TiDB namespace changed reads a covering index entry, not the whole namespace row. That cut TiKV traffic per unchanged read from 1.74 MB to 4.76 KB.
File bytes live in immutable blocks, cached in bounded RAM and disk tiers in front of the object store. Uploads stream their blocks to storage as they arrive, up to 5 GiB per file.
Drives that can accept an acknowledgement before durability can use writeback: mutations are applied in memory and published later, with p99 under 1 ms even when forwarded.
Go deeper