High availability on European cloud providers

Why this test#

While developing Flui, an open-source platform for deploying and managing the application lifecycle on one’s own servers or with European providers, I asked myself how to approach high availability. Specifically, the type of resilience I’m trying to test is the total failure of a data center.

The textbook answer is simple: three availability zones, a Kubernetes cluster deployed across them, and a replicated database. However, I wanted to know what actually happens when a zone goes down, and I wanted to measure this across European providers: these are features taken for granted on hyperscalers which hide real complexity. There are various reasons for adopting these providers: the push from regulations such as DORA (and the CADA, currently under discussion), concerns about data sovereignty, or simply economic considerations. Here I focus on the technical side.

So I set up the same K3s cluster (a lightweight Kubernetes distribution) on OVHcloud and Scaleway in Milan, on Hetzner in three cities, and on AWS, also in Milan. Same scripts, same tests, three repetitions for each failure.

A note on AWS. It’s my baseline: I’ve worked with it for years and know what it delivers. This isn’t a direct benchmark, because the conditions are different: non-identical machines, different vCPU types, different disks. But this is what the market offers, and the comparison must be made, cost included. Every service has its own characteristics, and everyone chooses what they need. I’ll line up the numbers side by side and explain what each difference is due to. I’m treating Hetzner separately because it doesn’t have availability zones but rather data centers located in different cities, and a direct comparison wouldn’t have made sense in this case.

This is the first part, covering the cluster. Databases, volumes, and other aspects worth discussing will follow.

In short#

What I built#

The cluster uses K3s: it’s a single executable that already contains everything needed, so it’s well-suited for small clusters like this one.

Three machines, one per zone, each with a node that acts as both a control plane and a worker. No managed services: no EKS, no managed databases. K3s is installed with the default settings; the only change I made is to set up three copies of the internal DNS, and I’ll explain why later. The scripts are public, and each cycle costs just a few euros.

Kubernetes stores the entire cluster state in a small database called etcd: which apps are running, where, and with what settings. Here, one copy runs per zone, and the three copies coordinate using a mechanism called Raft. In short, it works like this:

• a single copy acts as the leader: it receives writes and passes them on to the other two;
• a write is confirmed when it has a majority, that is, two out of three copies. This is why three copies can withstand the loss of one, but not two;
• the leader continuously sends a heartbeat signal to the other copies. If they stop receiving it for a few seconds, they elect a new leader, and here too a majority of votes is required;
• until a new leader is elected, the cluster’s state cannot be changed. Apps that are already running continue to respond.

I chose the 2 vCPU, 4 GB machine that every provider sells in all three zones in Milan: sufficient for our experiments at an affordable cost.

	AWS	OVHcloud	Scaleway
Where	Milan, 3 zones	Milan, 3 zones	Milan, 3 zones
Machine	c7i.large	c3-4	BASIC3-X2C-4G
CPU	Intel Xeon 8488C	AMD EPYC Milan	AMD EPYC 9555P
vCPU / RAM	2 / 4 GB	2 / 4 GB	2 / 4 GB
vCPU type	1 core with 2 threads, not shared	guaranteed resources, according to OVH	shared
Measured CPU steal	below 0.2%	below 0.1%	from about 1% to 13%, depending on the node
Disk	network gp3, 5,000 IOPS, 2,000 of them paid extra	local NVMe, not replicated	network, 5,000 IOPS, replicated in 3 copies

CPU. On AWS, the 2 vCPUs consist of one physical core with two threads, not shared with other customers (AWS). OVH guarantees resources for the family to which the c3 instance belongs (OVHcloud). Scaleway BASIC3 has shared vCPUs (Scaleway), and in Milan, dedicated vCPUs aren’t available in all three zones: for a three-zone cluster, the shared option is the only choice. That’s why I chose it.

Steal. I measured steal on all three during the same tests. Steal is the time the vCPU wants to run but the physical server has given the core to another machine. On AWS and OVH, it’s practically zero. On Scaleway, it ranges from about 1% to 13%, depending on the node and the workload.

Disks. OVH uses a local NVMe drive, which the pricing sheet refers to as “ephemeral”: it is not replicated (pricing sheet). This is fine for etcd, since it already maintains three copies of the data, one per zone; for a database, it’s a different story, and we’ll explore that in a dedicated experiment. Scaleway uses a network-attached disk replicated in three copies (Scaleway). AWS uses gp3, which includes 3,000 IOPS (AWS): to ensure a level playing field with Scaleway, which has 5,000 IOPS disks, I paid for 2,000 extra IOPS.

The Milan zones: network, disk and writes#

A note on this section: the performance differences described below are significant with high data volumes or under database stress; with normal use, you won’t notice them. Failover behavior, however, affects everyone.

All three providers describe their zones as separate data centers: AWS, OVHcloud (and the Milan page), and Scaleway. Details regarding distances and independence are often general, not specific to Milan, so I’ll limit myself to what I’ve measured.

Network#

The first panel matters for etcd: every write operation waits for a response from another zone, so cross-zone latency is factored into every write. Here, it ranges from 0.5 to 1.4 ms across all three providers, and none stands out.

The second panel measures performance when moving large amounts of data, such as database replicas and backups. OVH achieves between 3.7 and 4.1 Gbit/s; Scaleway manages 0.36 Gbit/s, which is the stated limit for this instance size, not for its network. AWS achieves 4.96 Gbit/s across all three pairs, but this needs to be interpreted carefully: it’s the 5 Gbit/s cap that AWS imposes on a single data stream (AWS). The guaranteed bandwidth for this instance is 0.78 Gbit/s, with peaks of up to 12.5 Gbit/s that typically last between 5 and 60 minutes (AWS). My ten-minute test fell within the peak period, so this doesn’t indicate how well it handles a long-duration transfer.

On AWS, cross-zone traffic is charged at $0.01 per GB in each direction, approximately 0.9 euro cents (price list). OVH does not charge for traffic on its private network (OVHcloud).

Disk#

This is the metric that matters most for etcd, because etcd does not acknowledge a write until the disk confirms it has been saved. OVH does 0.34 ms, with a p99 of 0.43: local NVMe, and it shows. AWS hovers around 3 ms, and even the slowest writes stay close to that, with a p99 of 3.4: it’s the most consistent network disk I’ve measured. In the typical case, Scaleway is faster than AWS at 1.1 ms, but one in every hundred writes takes 6-9 ms, and from etcd’s perspective, it reaches 12 ms.

However, these numbers don’t mean one disk is better than the other: they’re two different things. OVH uses a local disk attached to the physical server: extremely fast, but the pricing sheet calls it “ephemeral”; it isn’t replicated, and if the server fails, the data on that disk isn’t guaranteed. Scaleway and AWS use network-attached disks, which sacrifice some speed but offer other benefits:

• the data survives the machine: if the physical server fails, the disk is attached to a new machine in the same zone;
• they are replicated: Scaleway keeps three copies of all data (Scaleway), and AWS replicates the data from each disk across multiple servers in the same zone (AWS);
• they can be detached and moved: you can reattach them to another machine, scale them up, or save them with a snapshot, without touching the server;

For etcd, which already maintains three copies across three zones, the fast local disk is the ideal choice. For a database with a single copy, or for app data, the network disk can save you when the machine fails. It depends on what you store on it. OVH also sells network disks separately, which I haven’t measured here. And neither survives the loss of the entire zone.

Writes on Kubernetes#

Here you can see everything together. A write operation takes 2.8-3.6 ms on OVH, 3.4-5.0 ms on AWS, and 4.9-6.3 ms on Scaleway. The p99 is under 9 ms on AWS and OVH. On Scaleway, it’s 18-19 ms, except in one test, the one with the leader in it-mil-1, where it reaches 97 ms. It doesn’t depend on the zone: in another run, the same test stopped at 16 ms. In that minute, slow writes were simply more frequent.

The interesting detail is the contrast with the disk performance graph: in the typical case, Scaleway has a faster disk than AWS, but Kubernetes takes longer to confirm a write. So the disk alone doesn’t explain it.

To understand where Scaleway’s slow writes come from, I ran an additional round of tests, only there, measuring the steal and the disk every 100 ms. The result: it’s both, playing different roles. When the node receiving the writes has to wait for the CPU, because the physical server is giving it to another machine, the probability of a slow write rises from 0.35% to about 9%. The spikes above 100 ms, on the other hand, come from the network disk, which occasionally stalls for tens of milliseconds. When both issues occur simultaneously, more than one in four writes is slow. The node receiving the writes is the critical factor, while the leader node matters much less; and with the added traffic from the test app, slow writes increased by about half. With small, shared machines, it’s therefore best not to load that node with other work.

How I simulated the loss of a zone#

I didn’t shut anything down: I isolated a zone from the network using nftables. Its nodes remain powered on, but they don’t communicate with anyone, and no one communicates with them.

This is sufficient to simulate the loss of a zone: for the rest of the cluster, a shut-down zone and an unreachable zone are the same thing. In both cases all you get is silence, and neither etcd nor Kubernetes can distinguish between them. In fact, isolation is the most challenging scenario, because the isolated nodes are alive and continue to try to operate.

What it doesn’t simulate: the provider’s services in that zone, namely network disks, APIs and load balancers, remain up. And I don’t test partial failures, such as a slow zone or one that drops packets, which are a different type of failure. During isolation, only the SSH port to my address remains open so I can intervene; the cluster doesn’t use it, and the rule removes itself automatically when it expires.

For the second failure, I simulated a crash: the leader node suddenly reboots without writing anything to disk. Like a server losing power.

During the failure, a probe continuously writes to the Kubernetes API from a zone I don’t touch, with a 2-second write timeout, and measures how long writes stay blocked. It measures only the writes.

Meanwhile, a test app is running: a small web server with three replicas, one per zone, accessible via a single internal address. From a healthy node, a client continuously calls them, one request at a time, and marks any request that doesn’t respond within 2 seconds as failed. I forced one replica per zone, and that matters later: that’s why the lost replica doesn’t restart in another zone.

The zone remains isolated for 7 minutes because Kubernetes gives up on a lost node after about 6 minutes, and I wanted to see that happen. I repeated each failure three times per provider. Everything else is left at its defaults, because that is what someone installing K3s today would get.

The cluster survives, and the timings are the same everywhere#

On all three providers in Milan, as well as on Hetzner, the cluster loses a zone and continues to function, three tests out of three. The graph shows three different tests, from top to bottom:

• leader zone isolated: I isolate the zone where the leader is located. The other two copies of etcd can no longer communicate with it; they wait and then elect a new leader.
• leader crash: the leader node suddenly restarts. Here, too, the remaining copies must elect a new leader.
• zone without leader: I isolate a zone where the leader is absent. The leader still holds the majority, two copies out of three, and continues to operate: no election takes place. The pauses range from 0.03 to 0.07 seconds, which is why the data points are all close to zero.

When an election is needed, Kubernetes does not accept writes for 6-10 seconds, then resumes on its own, always with a single leader change. This is the time etcd waits before considering the leader lost and electing a new one. K3s sets this time longer than normal to accommodate slow networks, and etcd intentionally varies it slightly with each election. This is why the same provider shows 6 seconds in one test and 10 in the next. The values are always 6.1, 8.1, or 10.1 because my probe retries every 2 seconds.

So there is no ranking to be made. Out of 30 failures across all providers, including Hetzner, the pauses split as chance predicts: 13, 13, and 4 times, against the expected 15, 12, and 3. The details are in the repository.

These times can be reduced, but with slower networks, spurious elections become more frequent. I left the default settings because that’s what anyone installing K3s will encounter. And Scaleway’s shared vCPUs have nothing to do with this: etcd determines the times, not the machine.

What your app sees#

The fact that the cluster survives is good news for those who manage it. But anyone running an app on it wants to know something else: what users see.

There are two delays that matter. Kubernetes takes about one minute to stop sending traffic to a failed zone, and about 6 minutes to rebuild elsewhere what was there. An app with three replicas only experiences the first. A component that exists in a single copy experiences the second.

The replicas in healthy zones continue to respond. But until Kubernetes marks the node as lost, between 43 and 78 seconds after isolation, it continues to send requests to the failed replica as well. In my tests, during that window, between a quarter and half of the requests failed, about one in three, and each failed only after waiting the 2-second timeout. It’s not the app that goes down, but for that minute, one in three requests hangs and then fails. I haven’t measured it, but the simplest fix lies with the client: a short timeout and a retry, which will likely hit a healthy replica.

After about 6 minutes, between 344 and 379 seconds, Kubernetes gives up on the node and deletes the lost replica. But it doesn’t recreate it elsewhere: I’ve set it to one replica per zone, so it waits for the zone to come back online and recreates it there. When the zone comes back online, everything realigns within a minute and a half.

What K3s installs as a single copy#

There’s one thing I had taken for granted: that a cluster spanning three zones would be highly available in all its parts. That’s not the case: K3s deploys three components as a single instance (CoreDNS, Traefik, and metrics-server). If the lost zone is the one where they’re running, the cluster holds up but they don’t, until Kubernetes recreates them elsewhere after those 6 minutes, or until the zone comes back.

Component	What stops working	For how long
Traefik, the ingress	the app with three replicas is not reachable from outside	352-376 s in Milan, up to 379 s on Hetzner; checked from inside the cluster, from a healthy node
metrics-server	kubectl top, and pod autoscaling is left without data	395-409 s in Milan, from 372 s on Hetzner
CoreDNS, the internal DNS	no names in the cluster: anything that reconnects by name waits	until the zone returns or it is recreated elsewhere; measured in earlier runs, before the fix

In practice: three replicas across three zones are useless if the only ingress is in the lost zone. The app is up and running, but no one can reach it from the outside for 6 minutes.

The DNS issue came to light during database testing: when the zone running the DNS was lost, a replicated database remained down until the zone came back online, instead of restarting in just over a minute. I’ll come back to this when I discuss databases. That’s why, in all the tests in this article, CoreDNS runs in three replicas.

High availability must also be ensured for components you didn’t install yourself. One zone can become more important than the others without anyone having decided it.

Hetzner: three cities are not three zones#

Hetzner does not have availability zones within a single data center, and as far as I can tell, it has not announced any plans to implement them (Locations, What’s New). To have three “zones,” I used its three European data centers: Falkenstein and Nuremberg in Germany, and Helsinki in Finland. There are no other locations in Europe, so there wasn’t a closer option. The comparison with Milan isn’t on equal footing, which is why Hetzner stands apart.

Two machines: cpx22, with 2 shared vCPUs and 4 GB of RAM, the same size as in Milan; and ccx13, with 2 dedicated vCPUs and 4 GB of RAM, as a control. Why the cpx22 and not the CX23, which costs less: The Cost-Optimized series is available in limited quantities; during my tests, the CX23 was unavailable. Even if I had waited for it to become available, it would have made it very difficult to replicate the experiment given how infrequently it’s available across all three zones. The CPX series appears to be Hetzner’s standard offering, running on next-generation hardware (Hetzner).

The cluster performs as it does on the others, with the same 6-10 seconds, and the app also behaves as it does in Milan: traffic rerouted in 44-73 seconds, lost replica removed after about 6 minutes. The dedicated vCPUs do not affect the failure times, because those are determined by etcd.

You buy the machine, but not the distance. At the two German locations, the ccx13’s disk is about four times faster than the cpx22’s: 0.27 ms versus 1.1-1.2 ms. Even the slowest writes on Kubernetes, the p99, drop to one-third: 7-9 ms instead of 21-23 ms. In Helsinki, however, the ccx13’s disk performs like the cpx22’s, at about 1 ms. But with the leader in Helsinki, every write takes 63 ms on both machines, 6 to 12 times the normal time.

The reason lies in the majority rule. A write is confirmed when two out of three copies have it. With the leader in Germany, the other German node is sufficient, at 3 ms. With the leader in Helsinki, a node in Germany is required, at 24-36 ms. After a failure or a reboot, the leader may end up in Helsinki, and there’s no way to prevent it: it can only be moved manually.

On Hetzner, moreover, a single test isn’t enough. Three cpx22 clusters created on the same day yielded bandwidths between Falkenstein and Nuremberg of 4.9, 0.26, and 1.6 Gbit/s: every new cluster lands on different physical servers.

List prices for new orders, as of October 30, 2026, excluding VAT: cpx22 at €19.49 and ccx13 at €42.99 per month, plus €0.50 per IPv4 address per server (source). These apply to new orders since June 2026; servers activated earlier may pay less. Internal traffic is free (Hetzner). With the cpx22, the cluster costs about €60 per month, roughly half the price of the others. With the ccx13, it’s about €130, more or less the same as OVH in Milan: “Hetzner costs half as much” applies only to shared vCPUs.

What three zones cost#

The true cost of three zones lies in the price of the machines: latency between zones adds only a few milliseconds to write operations.

3-node cluster	Per month
AWS, c7i.large	about €259
OVHcloud, c3-4	about €122
Scaleway, BASIC3-X2C-4G	about €109

On-demand list prices as of October 30, 2026, excluding VAT, with disk and IPv4 included. AWS prices are converted from dollars to euros. On OVH in Milan, the local disk and IP are billed separately from the instance and are included in the total.

AWS costs a little more than double: about 2.1 times OVH and 2.4 times Scaleway. Additionally, on AWS you pay for cross-zone traffic, which is included on OVH.

What you get with the price difference on AWS: the most consistent network disk I’ve measured; non-shared vCPUs; and everything I didn’t use: EKS, RDS, and managed services. If you use them, you’re also paying not to build by hand what this article describes. Bandwidth, on the other hand, I wouldn’t count as a definite advantage: the guaranteed bandwidth for this instance is 0.78 Gbit/s, and my 4.96 falls within the peak range.

Two caveats. These are list prices, without any discount plans. And another family of machines on AWS, such as t3 or Graviton, would change the ratio.

Mistakes made while measuring#

Before publishing, I found and corrected three errors.

1. The probe was also measuring the connection. It opened a new TLS connection with every write, and writes to Kubernetes were 1.5-3 ms slower across all providers. I corrected and reran everything.
2. The DNS was in a single replica, the story told above. I noticed this because some tests took twice as long as others; I kept only the data with three replicas.

What this article does not cover#

• Three repetitions per provider: these demonstrate the behavior; they are not statistics.
• The probe measures at intervals of about 2 seconds, and only writes: I didn’t measure reads during the election.
• This isn’t a true zone failure: I isolate the nodes, but network disks, the provider’s API, and the load balancer remain up. I didn’t test partial failures, such as a slow zone or one that drops packets.
• Small machines, no real load during the failure.
• On AWS, only machines: no EKS or RDS.
• Outside of this scope: the database, app volumes, and how traffic arrives from the outside, that is, load balancers and public DNS, which I’ll address with dedicated experiments.

Conclusions#

Where a provider offers three nearby zones, there’s no reason to forgo high availability: it all depends on the software built on top of it. On OVH and Scaleway in Milan, the cluster withstood the loss of one zone in all tests, just as it did on AWS. If the leader is in the lost zone, writes stop for 6-10 seconds, the same across all providers; otherwise, there is barely any pause.

The differences lie elsewhere: in the disk, the CPU, and the price, which must be chosen with a clear understanding of what you’re buying. On Hetzner, where the zones are three cities, and with the leader in Helsinki, you pay the price for the distance with every write. And in the components that K3s installs as a single copy, which make any given zone the one you simply cannot afford to lose.

For Flui, this moves the problem. Three zones across European providers are a viable option, and the cluster stands on its own. High availability depends on what’s above it: the system components, which must have a copy in every zone, and then the databases and volumes, which must survive the loss of a zone. This is the work Flui must do on behalf of its users.

The next step is the database itself: Postgres across three zones, how much it costs not to lose a commit and how long it stays down.

The scripts and data are in the dawit-io/ha-eu-bench repository, along with write-by-write data for every run and the source of every figure quoted.

Flui is at flui.cloud, and its code is on GitHub.