How long it takes to move an IP address between two servers

A reserved IP is DigitalOcean's version of a floating IP: a public address you can move from one Droplet to another with a single API call. The usual reason to have one is failover. Two servers, one address, and when the first server has a problem you point the address at the second. I wanted to know what that move looks like from the client's side: how long it takes, whether requests get lost, and what happens to a connection that's already open. So I moved one back and forth 20 times with traffic on it and logged every request.

The short answer: traffic moved about 4.2 seconds after the API call, with almost no gap. New connections barely noticed. Every connection that was already open got reset, every time.

The setup

• Two Droplets in fra1, A and B, the smallest size, Ubuntu 24.04. Each ran the same small Python server: HTTP on port 8080 answering with its own hostname, and a line-echo service on port 9000 that also prefixes its hostname.
• One reserved IP in fra1, starting on A.
• Two clients, each sending traffic to the reserved IP for the whole run: a third Droplet in ams3, and my laptop in Bulgaria. On each, one thread opened a new TCP connection and made an HTTP request every 100 ms, with a 1-second timeout. Another thread kept one TCP connection open and sent a line every 100 ms, reconnecting if anything went wrong.
• A script on my laptop waited 40 seconds, then called the API to move the IP to the other Droplet, then polled the action until DigitalOcean reported it done. It repeated that 20 times, A to B, B to A, and so on.

The moves ran from 07:23 to 07:38 UTC. The two clients made 57,953 requests between them. Because every answer names the server that sent it, I could see exactly when traffic switched.

How long a move takes

All times are seconds after my script sent the API request.

Two things stood out.

The two clients switched at the same moment, within about 30 ms of each other, even though one was in Amsterdam and the other was on a home connection in Bulgaria. So the switch happens at one place in DigitalOcean's network, not separately along each route.

And traffic moved before the API said the move was done, in all 20 moves, by about 2.4 seconds. I polled the action every half second, so some of that is my polling, but not two seconds of it. If your failover script waits for completed before it does anything else, traffic is already on the new server by then.

There was no flapping. Once the new Droplet answered a request, the old one never answered another, in any of the 80 combinations of move, client and connection type.

New connections

The gap between the last answer from the old Droplet and the first answer from the new one was a median of 0.05 seconds from the laptop and 0.08 seconds from ams3, at most 0.18 seconds. My probes ran every 100 ms, so that's within one probe. Not one new-connection request timed out because of a move.

A few did fail, all the same way, with a connection reset:

Each of those was a single request, never two in a row. My reading is that these were requests in flight at the moment of the switch: the connection was opened on the old Droplet, the next packets arrived at the new one, and the new one, having no idea about that connection, answered with a reset. A request that takes longer is more likely to be caught mid-flight, which would explain why the laptop, with three times the round trip, was hit four times as often. I didn't capture packets, so that's an inference from timing, not something I watched happen.

Connections that were already open

This part was not subtle. The long-lived connection was reset on every move, on both clients, 40 out of 40. On ams3 all 20 were ConnectionResetError. On the laptop it was 14 resets and 6 BrokenPipeError, which is the same thing noticed while writing instead of reading.

The reset arrived together with the switch, a median of about 20 ms after the first new connection reached the new Droplet. The client then reconnected and landed on the new Droplet 0.18 to 0.19 seconds after its last good reply, which is just my client's 100 ms pause plus a new connection. So for a client that reconnects straight away, the cost of a move is one failed request.

It makes sense once you think about where the connection lives. Its state is in the old Droplet's kernel. The new Droplet has never seen it, so when packets for it show up, a reset is the correct response. A reserved IP moves an address, not connections. Anything that holds a connection open, a database pool, a WebSocket, a gRPC channel, will have to reconnect after a move, and it'll find out the moment the move happens.

What I got wrong

My first local test of the client showed every new-connection request timing out. I started debugging the client. The client was fine. Python's HTTPServer calls socket.getfqdn() when it starts, and on my Mac that lookup took several seconds, so the server simply wasn't listening yet. I overrode server_bind to skip it. While chasing that I also left a test server running from an earlier attempt, which kept answering on the port and made the next test confusing. Before testing a network client, check what is actually listening.

The cleanup didn't go as planned either. My script deleted the three Droplets, then released the reserved IP. The release call returned success, but the IP was still in my account afterwards, unassigned. Its action history showed an automatic unassign that failed, then one that completed a few seconds later. After that every delete returned 422 The reserved IP action is unprocessable for about five minutes. The one at 07:52:47 went through, and the IP was gone at 07:53. I think I caused the race myself: deleting a Droplet makes DigitalOcean unassign its reserved IP, and my release call arrived while that was still going on. The order that makes sense is to release the IP first and delete the Droplets after, and I've changed the script to do that. The five unassigned minutes cost nothing, because a reserved IP isn't billed until it has run up $1, but the first success response clearly wasn't the end of it.

Caveats

One region, one morning, 20 moves, and both Droplets healthy. In a real failover the old server is usually in trouble, and I didn't test that. The clients only sent TCP. My probes were 100 ms apart, so anything shorter than that I can't see. Switch times are measured against my laptop's clock, and the ams3 client's clock was NTP-synced but not compared against the laptop's, so treat the 30 ms agreement between clients as approximate.

My laptop also logged two timeouts, a second apart, two to three seconds before one of the moves, while traffic was steady on the old Droplet. The ams3 client saw nothing at that moment, so I put those down to my home connection, not the reserved IP, and left them out of the counts above.

It cost almost nothing. Three of the smallest Droplets ran for 27 minutes, and DigitalOcean doesn't charge for a reserved IP while it's assigned to a Droplet.

What I'd take from it

• Expect a move to take about 4 seconds from the API call, not the 6 to 7 it takes for the action to say completed.
• New connections will mostly be fine. A request in flight at the moment of the switch can be reset, so retry once on a reset.
• Every open connection will be reset. Make sure your clients reconnect on their own, quickly, and without a long backoff on the first try.
• If you're building failover on top of this, the move itself is the fast part. Most of your downtime will be however long it takes you to decide that the first server is down.
• When you tear down a test, check that the reserved IP is actually gone. An unassigned one is billed.