Routing Asia-Pacific Video Traffic With etcd-Backed Region Config

Last spring I shipped a bug that only existed in Seoul. A viewer in South Korea opened the trending page on TopVideoHub and got the Tokyo feed — Japanese titles, JP category weighting, the wrong CJK tokenizer bucket. Meanwhile Osaka was fine. The code was identical on every node. The config was not.

We run region routers as a small fleet of LiteSpeed + PHP 8.4 boxes behind Cloudflare. Each router decides, for an incoming request, which trending pool to serve, which language pack to load, and which SQLite FTS5 tokenizer profile to attach for search. For a long time that decision was driven by a regions.json file baked into each deploy. Someone (me) edited the KR block on one box during an incident, fixed the symptom, and never propagated it. Config drift did the rest. Multiply that across eight countries and three languages and you get a system where nobody can answer the question "what is the actual routing table serving traffic right now?"

This post is how we replaced the baked-in JSON with TopVideoHub running a single source of truth in etcd, and the specific patterns — watches, leases, and compare-and-swap transactions — that made it safe to change region routing without a redeploy and without waking anyone up.

The problem in concrete terms

A region router needs three things per country to serve an Asia-Pacific audience:

• Trending pool — which ingest bucket to read (e.g. pool:kr-music, pool:jp-general). Editors retune these during big events.
• Language + tokenizer — Korean traffic must load the Hangul-aware FTS5 profile, not the CJK-generic one, or search recall drops hard.
• Fallback chain — if the KR pool is stale (ingest lagging), fall back to pool:apac-general before falling back to global.

Those values change for editorial reasons, not code reasons. A K-pop comeback means we want to boost kr-music for six hours. An ingest outage in Taiwan means we flip TW's fallback. None of that should require a Git commit, a build, and an FTP deploy to four hosts. It should be a config write that every router sees within a second or two, atomically, with an audit trail of who changed what.

That is exactly the shape etcd is built for.

Why etcd and not a database or a config file

I evaluated three options and I want to be honest about the trade-offs rather than sell you etcd blindly.

• A row in SQLite/MySQL. We already run SQLite everywhere. But there is no push. Routers would have to poll, and polling either wastes queries or adds latency to change propagation. There is also no clean way to do a compare-and-swap across a fleet without inventing a locking scheme.
• Consul. Genuinely good and arguably friendlier for service discovery. We didn't need the service-mesh half, and I already run etcd for another internal tool, so operationally it was one less moving part.
• etcd. Gives me a strongly-consistent key-value store with a real watch primitive (server pushes changes over a stream), leases (keys that auto-expire — perfect for temporary boosts), and transactions with compare-and-swap on a key's revision. That combination is what config distribution actually needs.

The deciding factor was the watch. I want a router to learn about a routing change by being told, not by asking. etcd's revision-based watch means a client can reconnect after a network blip and resume from the last revision it saw, so it never misses an update in the gap.

Modeling the routing table as keys

I keep the key layout boring and hierarchical so I can watch a whole prefix at once:

/topvideohub/regions/KR/pool        -> "kr-music"
/topvideohub/regions/KR/lang        -> "ko"
/topvideohub/regions/KR/tokenizer   -> "fts5_hangul"
/topvideohub/regions/KR/fallback    -> "apac-general,global"
/topvideohub/regions/JP/pool        -> "jp-general"
...

Everything under /topvideohub/regions/ is the routing table. A router watches that single prefix. A temporary boost gets written under the same prefix but attached to a lease so it cleans itself up. Because etcd stores plain bytes, I keep values as short scalars rather than blobs of JSON — it makes compare-and-swap meaningful per-field and keeps watch events small.

Reading a key from PHP over the v3 JSON gateway

PHP has no first-class etcd client I trust for production, but I don't need one. etcd v3 exposes its gRPC API over a JSON/HTTP gateway, so a plain curl call does the job. Keys and values travel base64-encoded. Here's the primitive I build everything on:

<?php
declare(strict_types=1);

final class EtcdClient
{
    public function __construct(
        private readonly string $endpoint = 'http://127.0.0.1:2379',
        private readonly float $timeout = 1.0,
    ) {}

    /** Range read: single key, or a prefix when $prefix = true. */
    public function get(string $key, bool $prefix = false): array
    {
        $body = ['key' => base64_encode($key)];
        if ($prefix) {
            // range_end = key with last byte incremented => prefix scan
            $end = substr($key, 0, -1) . chr(ord(substr($key, -1)) + 1);
            $body['range_end'] = base64_encode($end);
        }

        $ch = curl_init($this->endpoint . '/v3/kv/range');
        curl_setopt_array($ch, [
            CURLOPT_POST => true,
            CURLOPT_POSTFIELDS => json_encode($body, JSON_THROW_ON_ERROR),
            CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT => (int) ceil($this->timeout),
        ]);
        $raw = curl_exec($ch);
        if ($raw === false) {
            throw new RuntimeException('etcd unreachable: ' . curl_error($ch));
        }
        curl_close($ch);

        $decoded = json_decode($raw, true, 512, JSON_THROW_ON_ERROR);
        $out = [];
        foreach ($decoded['kvs'] ?? [] as $kv) {
            $out[base64_decode($kv['key'])] = base64_decode($kv['value']);
        }
        return $out;
    }
}

That's the whole dependency surface: curl and json. It works the same on LiteSpeed as it does on CLI, and it fails fast with a one-second timeout so a slow etcd never blocks a page render — the hot path never calls this directly anyway, as you'll see below.

Writing config safely with leases and compare-and-swap

The write side is where correctness lives. Two rules I refuse to break:

1. A temporary boost must expire on its own. If the editor forgets to revert it, or the process that set it dies, the boost must vanish. That is a lease.
2. A write must not silently clobber a concurrent write. If two people retune KR at once, one of them must lose loudly, not silently. That is compare-and-swap on mod_revision.

I wrote the writer in Python because that's where our editorial tooling lives. It grants a lease, then does a transactional put guarded on the field's current revision:

import base64
import requests

ETCD = "http://127.0.0.1:2379"

def b64(s: str) -> str:
    return base64.b64encode(s.encode()).decode()

def grant_lease(ttl_seconds: int) -> int:
    r = requests.post(f"{ETCD}/v3/lease/grant", json={"TTL": ttl_seconds}, timeout=2)
    r.raise_for_status()
    return int(r.json()["ID"])

def current_revision(key: str) -> int:
    r = requests.post(f"{ETCD}/v3/kv/range", json={"key": b64(key)}, timeout=2)
    r.raise_for_status()
    kvs = r.json().get("kvs", [])
    return int(kvs[0]["mod_revision"]) if kvs else 0

def cas_put(key: str, value: str, expected_rev: int, lease: int = 0) -> bool:
    """Put only if the key is still at expected_rev. Returns False on conflict."""
    put = {"key": b64(key), "value": b64(value)}
    if lease:
        put["lease"] = lease
    txn = {
        "compare": [{
            "key": b64(key),
            "target": "MOD",
            "result": "EQUAL",
            "mod_revision": str(expected_rev),
        }],
        "success": [{"request_put": put}],
        "failure": [],
    }
    r = requests.post(f"{ETCD}/v3/kv/txn", json=txn, timeout=2)
    r.raise_for_status()
    return r.json().get("succeeded", False)

# Boost KR music for 6 hours, but only if nobody changed it under us.
key = "/topvideohub/regions/KR/pool"
rev = current_revision(key)
lease = grant_lease(6 * 3600)
if cas_put(key, "kr-music", expected_rev=rev, lease=lease):
    print("KR boosted to kr-music for 6h")
else:
    print("conflict: someone else changed KR/pool, re-read and retry")

The lease is the part I love. After six hours etcd deletes the key on its own, which fires a DELETE watch event on every router, which triggers the routers to fall back to their default pool. No cron job to revert the boost. No sticky config. The expiry is the revert, and it's driven by the store, not by any single machine staying alive.

When we need a boost to outlive its lease we send a keep-alive; when we want it to end early we revoke the lease. Either way there is exactly one mechanism.

A Go sidecar that turns watches into local state

I do not want every PHP worker holding an open watch stream — that's thousands of streams and a lot of reconnection logic in a language that's awkward for long-lived connections. Instead each router runs one tiny Go sidecar. It holds a single watch on the /topvideohub/regions/ prefix, and on every change it rewrites a local snapshot and bumps APCu. PHP reads the snapshot; the sidecar owns the connection.

package main

import (
    "context"
    "encoding/json"
    "log"
    "os"
    "strings"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
)

const prefix = "/topvideohub/regions/"

func main() {
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"127.0.0.1:2379"},
        DialTimeout: 5 * time.Second,
    })
    if err != nil {
        log.Fatalf("connect: %v", err)
    }
    defer cli.Close()

    table := map[string]string{}

    // 1) Load the current state and remember the revision we read at.
    resp, err := cli.Get(context.Background(), prefix, clientv3.WithPrefix())
    if err != nil {
        log.Fatalf("initial get: %v", err)
    }
    for _, kv := range resp.Kvs {
        table[strings.TrimPrefix(string(kv.Key), prefix)] = string(kv.Value)
    }
    writeSnapshot(table)

    // 2) Watch from the NEXT revision so we never miss or double-apply an event.
    wch := cli.Watch(context.Background(), prefix,
        clientv3.WithPrefix(),
        clientv3.WithRev(resp.Header.Revision+1),
    )
    for wr := range wch {
        if err := wr.Err(); err != nil {
            log.Printf("watch error, client auto-reconnects: %v", err)
            continue
        }
        for _, ev := range wr.Events {
            field := strings.TrimPrefix(string(ev.Kv.Key), prefix)
            switch ev.Type.String() {
            case "PUT":
                table[field] = string(ev.Kv.Value)
            case "DELETE": // e.g. a lease expired
                delete(table, field)
            }
        }
        writeSnapshot(table)
        log.Printf("applied %d event(s), %d keys live", len(wr.Events), len(table))
    }
}

func writeSnapshot(table map[string]string) {
    blob, _ := json.Marshal(table)
    // Atomic replace: write temp then rename so PHP never reads a half file.
    tmp := "/dev/shm/regions.json.tmp"
    if err := os.WriteFile(tmp, blob, 0o644); err != nil {
        log.Printf("write temp: %v", err)
        return
    }
    if err := os.Rename(tmp, "/dev/shm/regions.json"); err != nil {
        log.Printf("rename: %v", err)
    }
}

Two details carry all the safety here:

• Watch from resp.Header.Revision + 1. The initial Get gave me a consistent snapshot at some revision R. Starting the watch at R+1 means I apply every subsequent change exactly once, with no gap and no replay. This is the single most important correctness property, and it's the thing polling can never give you.
• Atomic file replace. I write to /dev/shm (tmpfs — fast, and it never touches disk) and rename() into place. rename is atomic on the same filesystem, so a PHP worker either sees the old complete file or the new complete file, never a truncated one.

The etcd Go client handles reconnection internally. If the network drops, it re-establishes the watch and resumes from where the channel left off. My code just keeps ranging over the channel.

Reading config in the hot path

Now the part that runs on every single request. It must be fast and it must never depend on etcd being reachable at request time. It reads the tmpfs snapshot, caches the parse in APCu for a couple of seconds, and resolves the region with an explicit fallback chain:

<?php
declare(strict_types=1);

final class RegionRouter
{
    private const SNAPSHOT = '/dev/shm/regions.json';
    private const APCU_KEY  = 'regions:table';
    private const APCU_TTL   = 2; // seconds; sidecar pushes faster than this anyway

    private function table(): array
    {
        $cached = apcu_fetch(self::APCU_KEY, $ok);
        if ($ok) {
            return $cached;
        }
        $raw = @file_get_contents(self::SNAPSHOT);
        $table = $raw === false ? [] : (json_decode($raw, true) ?? []);
        apcu_store(self::APCU_KEY, $table, self::APCU_TTL);
        return $table;
    }

    /** @return array{pool:string, lang:string, tokenizer:string} */
    public function resolve(string $country): array
    {
        $t = $this->table();
        $get = static fn(string $c, string $f): ?string => $t["$c/$f"] ?? null;

        // Follow the fallback chain until a live pool answers.
        $chain = array_merge(
            [$country],
            explode(',', $get($country, 'fallback') ?? 'apac-general,global')
        );
        foreach ($chain as $c) {
            $c = trim($c);
            $pool = $get($c, 'pool');
            if ($pool !== null) {
                return [
                    'pool'      => $pool,
                    'lang'      => $get($c, 'lang') ?? 'en',
                    'tokenizer' => $get($c, 'tokenizer') ?? 'fts5_cjk',
                ];
            }
        }
        // Absolute last resort — never 500 a page over config.
        return ['pool' => 'global', 'lang' => 'en', 'tokenizer' => 'fts5_cjk'];
    }
}

// Usage inside a controller:
$router = new RegionRouter();
$cfg = $router->resolve($_SERVER['HTTP_CF_IPCOUNTRY'] ?? 'US');
// $cfg['tokenizer'] now selects the right FTS5 profile for search.

We take the country from Cloudflare's CF-IPCountry header, which is already on every request. The resolver reads from APCu, falling through to the tmpfs file, falling through to a hardcoded global default. There is no path where a page fails to render because etcd is down — the worst case is stale config for a few seconds, which is exactly the right trade for a video aggregator.

Failure modes we actually hit

Documenting the happy path is easy. Here's what actually bit us and how the design absorbed it.

• etcd unreachable at boot. If the sidecar can't do its initial Get, it exits and systemd restarts it. Until it succeeds, PHP keeps reading the last good /dev/shm/regions.json from the previous run. Config is stale but serving. I chose stale-and-serving over fresh-or-dead every time.
• A watch fell behind and got compacted. etcd compacts old revisions to reclaim space. If a sidecar was offline long enough that its resume revision was compacted, the watch returns a compaction error. The fix is to treat that error as "start over": re-Get the prefix and re-establish the watch at the new revision. Cheap, since the whole table is a few dozen tiny keys.
• Clock-free expiry. Early on I worried about boost expiry depending on synchronized clocks. It doesn't. Leases are managed by the etcd cluster's own clock, and expiry fires a real DELETE event. No router needs an accurate wall clock to revert a boost.
• The thundering-herd non-problem. Because only one sidecar per box watches etcd, a 40-worker LiteSpeed pool creates one watch stream, not forty. Scaling workers costs etcd nothing.

Conclusion

Moving region routing into etcd killed an entire class of incident for us. There is now exactly one place that answers "what is KR serving right now," every router learns about changes within a second through a pushed watch rather than a poll, temporary editorial boosts clean themselves up via leases, and concurrent edits collide loudly through compare-and-swap instead of silently clobbering each other. The PHP hot path never blocks on the network and never 500s over config, because the Go sidecar does the hard connection work and hands PHP an atomic tmpfs snapshot.

If you're running any kind of multi-region service where operational config changes for human reasons on a human timescale — feature flags, routing tables, per-market tuning — the etcd triad of watch, lease, and transaction is worth the small operational cost. Model your config as short flat keys under one prefix, push changes into local state with a tiny sidecar, and keep the request path reading from memory. The store becomes the source of truth; every node just mirrors it. Seoul has gotten the Seoul feed ever since.