Scaling and high availability
What happens when a server of a METAFILES site fails, what is automatic today, and how a site grows.
Every tier of a METAFILES site has at least two of everything: two app servers, two office servers, three storage servers, two public call relays and two tunnel connectors. This page says what happens when each part fails, and how the site grows. Where something needs a person today, it says so.
When something fails
| What fails | What happens | Automatic? |
|---|---|---|
| An app server | The web site address moves to the other app server, and the load balancer stops sending traffic to the failed one. Portals keep serving from the other server and sessions survive. In drills, the address moved within two seconds and no writes were lost. | Yes |
| A tunnel connector | The connector on the other app server carries all traffic. | Yes |
| A database member | The remaining member and a tie-breaker keep the database cluster running. Drills lost no writes. | Yes |
| The cache's primary | A replica is promoted. Drills measured a write gap of about 6 seconds, with sessions kept. | Yes |
| A storage server, disk or storage gateway | The cluster serves from the remaining two copies and heals itself; the other gateways take over. | Yes |
| Two storage servers | Writes pause until one returns; nothing is lost. | Yes, writes resume |
| A secret-vault node | The other two serve; the restarted node unseals itself. | Yes |
| An office server | The other office server serves Office and calls. | Yes |
| A public call relay | Clients use the other relay. | Yes |
| The active engine | Portals, sign-ins and calls stay up, but file reads and writes and Office saves pause. An operator approves a fenced failover in Site & Fleet, which locks out the old engine and promotes the standby. No acknowledged write is lost. | Approved by a person |
| The control plane | Users keep working. New builds, deletes and machine actions wait until it is back. | Users unaffected |
Why the engine failover is fenced. Two engines must never write at the same time. So the standby has no access to storage while it waits; a failover first removes the old engine's storage access and only then promotes the standby, and a fenced engine that reboots will not start. In tests, file access was back about 40 seconds after the outage began.
Coming: the next generation of the engine fails over automatically, runs on several app servers at once, and removes the per-organisation write lock.
How the site scales
Capacity grows at three levels:
| Level | What happens | Today |
|---|---|---|
| 1. Automatic, inside the existing servers | The storage cluster balances data across its disks by itself, and the load balancer spreads work across both app servers and all three storage gateways. Office starts a process per document; calls are spread per call. | Live |
| 2. A person plus a guided step | Monitoring shows a tier filling up; the MSP's operator has a server enlarged or a disk added, then the site is updated to use it. | Enlarging a server and adding storage disks are done as guided steps today. Spare machines can be added from the console. |
| 3. Fully automatic | A scaler watches the site and adds capacity within set limits, without a person. | Coming |
Coming: adding app, office and storage servers from the console, creating them through a hypervisor or cloud (Proxmox VE, libvirt/KVM, OpenStack, Amazon EC2), and an automatic scaler. Every automatic action will start in approve mode, where a person confirms each plan.
Watching the site
The control plane observes every machine every 10 minutes and after every action, and the results appear in Site & Fleet: machine health, the storage, database and cache clusters, the load balancer, site addresses, Office and Talk, certificate expiry and the call relays. Once a day it compares every server with the site's intended state and reports any difference. See Site and fleet.