The UDP Buffer Nobody Tunes: How a 256KB Kernel Default Cost Us 30% of Our Packets

The problem

A video platform I worked on wasn't crashing — it was worse than that. Calls stayed "up" but froze every few seconds, audio turned robotic, and it only happened between 5 and 7 PM. WebRTC sessions were logging close to 30% packet loss during that window. Network ops checked switches, routers, bandwidth — utilization sat around 40%, nothing congested. Ticket closed: "not a network problem."

It was. Just not in the place anyone was looking.

Why it happens

The instinct with UDP is to treat it like "TCP without the reliability guarantees." That framing hides the part that actually matters: TCP has backpressure built in. When a TCP receive buffer fills, the receiver shrinks its window and the sender slows down. UDP has none of that. The kernel gets a datagram, tries to place it in a receive buffer, and if there's no room, it silently drops it — no retry, no signal to the sender, nothing. From the app's point of view, the packet simply never existed.

tcpdump showed packets arriving cleanly at the NIC. Application logs showed holes in the sequence numbers. Packets were vanishing somewhere between the wire and the process — which meant the kernel, not the code, was the suspect.

The confirming metric was udp_receive_buffer_errors, sitting at zero off-peak and spiking into the thousands-per-second exactly when complaints rolled in. netstat -su made it explicit: 223,401 receive buffer errors out of ~752K packets received. The Ubuntu default for net.core.rmem_default was 212,992 bytes — roughly 256 KB. At peak, the service was pushing ~50,000 packets/sec at ~1,200 bytes each: about 60 MB/sec into a 256 KB bucket. That's roughly 4 milliseconds of headroom before the buffer fills. Get descheduled for longer than that — completely normal under load — and the kernel starts discarding.

It compounds, too: WebRTC detects the quality drop from the loss and adds redundancy and retries at the application layer, which throws more traffic at the same undersized buffer. The fix for congestion becomes the thing that deepens it.

What to do about it

The fix is two layers, and both are required — most teams only do one.

Kernel layer — raise the ceiling and the default:

# /etc/sysctl.conf
net.core.rmem_max = 16777216      # 16 MB max
net.core.rmem_default = 16777216  # 16 MB default for new sockets

sysctl -p

16 MB instead of 256 KB turns 4ms of headroom into roughly 260ms.

Application layer — the socket still has to ask for it, and you have to verify what it actually got:

int sock = socket(AF_INET, SOCK_DGRAM, 0);
int buffer_size = 16 * 1024 * 1024;

if (setsockopt(sock, SOL_SOCKET, SO_RCVBUF,
               &buffer_size, sizeof(buffer_size)) < 0) {
    perror("Failed to set socket receive buffer");
}

int actual_size;
socklen_t len = sizeof(actual_size);
getsockopt(sock, SOL_SOCKET, SO_RCVBUF, &actual_size, &len);
printf("Requested: %d bytes, Got: %d bytes\n", buffer_size, actual_size);

That getsockopt check matters more than it looks — some distros silently double what you request for internal bookkeeping, so "did I get what I asked for" is not a safe assumption.

One dead end worth naming: bumping the packet-processing thread's scheduling priority. It seems logical — drain the buffer faster — but it starves other threads of CPU time, and now you've traded packet loss for database timeouts. Buffer sizing is the correct first move; thread priority is a last resort, not a first instinct.

Key takeaways

• UDP has zero backpressure — a full receive buffer means silent, unrecoverable loss, not slowdown.
• Kernel socket buffer defaults (net.core.rmem_default) are generic and almost never sized for sustained high-throughput UDP.
• Fixing it requires both the sysctl change and an explicit setsockopt(SO_RCVBUF) call in your app — plus a getsockopt check to confirm what you actually got.
• Watch udp_receive_buffer_errors from netstat -su as a standing metric; it should be zero, and any sustained non-zero value is a real signal, not noise.
• Think in bursts, not averages — a tame req/sec number can still overflow a small buffer during a 200ms spike.

I wrote up the full debugging story — including the wrong turn I took first and what I now monitor because of it — on Medium.