Cilium Hubble on RKE2: Microservice Traffic Observability (Seeing the Invisible)

Part of the homelab build series. Every packet in my cluster already passes through Cilium's eBPF datapath on all 29 nodes. Hubble turns that into a live map of who talks to whom, with no sidecars and no code changes. This is how I switched it on for RKE2, the two traps that crash-looped it, and the uncomfortable thing I found when I went looking for the HTTP metrics.

The Hubble UI service map for one namespace, drawn live from real traffic.

Introduction

The map you already had and couldn't see

Ask anyone to draw their cluster's network and you'll get a confident whiteboard sketch that's quietly wrong.

The frontend talks to the API, the API talks to Postgres, everything talks to DNS. Sure. But which pod is really hammering that service? Was that "connection refused" a crashed pod, a missing endpoint, or a NetworkPolicy nobody remembers writing? The real traffic graph, the one packets actually follow, is invisible. So we debug it the old way: kubectl exec into a pod, run tcpdump, squint, and guess.

Here's the thing I'd missed for months. My CNI is Cilium. It's eBPF, it is the datapath, and it was already seeing every single packet on every node. I just wasn't looking.

Hubble is how you look. It turns Cilium's datapath into a stream of flows that say "kubeflow/notebook → kube-system/coredns:53, FORWARDED", not "10.42.3.7 → 10.42.1.9". No sidecar, no code change, and no app knows it's being watched.

So I switched it on. Within an afternoon I had a live service map behind Keycloak, hubble observe streaming every flow on the cluster, and network-policy decisions I could read instead of guess. It also cost me a week of debugging, and it taught me exactly where its vision stops.

What Cilium and Hubble actually are

Two layers, one on top of the other:

• Cilium is the CNI. It wires pod networking and enforces NetworkPolicy with eBPF programs in the kernel rather than iptables rules. Every workload gets a stable identity derived from its labels, not its IP, which is why its telemetry reads like Kubernetes instead of a wall of addresses. Here it's version 1.19.1, bundled with RKE2, using a VXLAN overlay alongside kube-proxy.
• Hubble is the observability built on top. Each cilium-agent exposes its flows on a socket. Hubble Relay gathers every node's stream into one cluster-wide feed. Hubble UI draws it as a live map.

You get three ways in: the CLI (hubble observe, grep for the network), the UI (the live service map), and Prometheus metrics (flow rates, drops, DNS).

The mental model I use: Hubble is tcpdump with Kubernetes identity, across the whole cluster, at almost no cost. It sees layers 3 and 4 for everything. Layer 7 is a different story, and I'll get to it.

How Hubble sees the cluster: cilium-agent on every node, Relay in the middle, the UI behind oauth2-proxy and Keycloak.

How eBPF sees a packet

"It's eBPF" gets said a lot and explained rarely, so here is what happens to one packet.

First, what it is. eBPF is a feature of the Linux kernel, not a product. Cilium is one of many tools built on it.

eBPF lets you load a small program into the running Linux kernel and attach it to a hook. The kernel checks the program before it runs, so it can't crash the node or loop forever. Cilium attaches its programs to the network hook on each pod's virtual interface, the veth pair that joins the pod to the node. Nothing can enter or leave a pod without passing one.

When a packet arrives, the program does three things, all inside the kernel:

1. It works out who is talking. It looks up the source and destination addresses in an eBPF map, a hash table shared between the kernel and the agent, and gets back two identities. An identity is a number that stands for a set of labels, so app=orders in one namespace is one identity however many pods carry it.
2. It decides. A second map holds the policy for that pod: which identities may reach which ports. One lookup gives the verdict, forward or drop. That is the difference from iptables, which walks a list of rules until one matches and so gets slower as policies grow.
3. It reports. The program writes a short event into a buffer the agent reads: the two identities, the ports, and the verdict. The packet itself is never copied out of the kernel.

Hubble is what reads those events. The agent turns the identities back into pod names and labels, and keeps the result as a flow. So the traffic flow you see in hubble observe is the same decision the kernel made, written down as it was made, not a second system guessing from the outside.

That is also why the service map draws itself. Cilium sees every connection, and it already knows which pod owns which address because it handed the addresses out. Nothing has to be registered or annotated: an edge appears on the map because traffic passed. The other side of that is that a service which hasn't talked recently has no edge.

Two details explain the cost. Cilium doesn't report every packet. By default it reports when a connection opens, changes state or closes, which is why a node pushing a lot of traffic still produces only a few hundred flows a second. And nothing is added to the pod: no sidecar, no extra network hop, no restart.

The same word, different tools. If you've used OpenTelemetry's eBPF instrumentation, this is not that. Both load programs into the kernel, but they attach in different places. Cilium hooks the pod's network interface and sees packets. OpenTelemetry's instrumentation hooks the application process and sees requests: method, path, status and duration, read before the application encrypts them. So one can block traffic and is blind inside TLS, and the other only observes but can see inside a service mesh. "It uses eBPF" tells you how a tool collects its data, not what data it collects.

That design also sets the limit. The kernel program reads addresses, ports and TCP flags. It does not read what is inside the connection. An HTTP path or a status code is layer 7, and for that Cilium has to hand the connection to a proxy.

Switching it on in RKE2

RKE2 ships Cilium as a bundled chart, so you don't install Cilium yourself. You layer values onto it with a HelmChartConfig called rke2-cilium in kube-system, and RKE2's helm controller merges and re-applies it within about 30 seconds. My whole Hubble setup is one file. Here's the heart of it:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium          # must match the bundled release
  namespace: kube-system
spec:
  valuesContent: |-
    hubble:
      enabled: true
      relay: { enabled: true, replicas: 1 }
      ui:    { enabled: true, replicas: 1, ingress: { enabled: false } }
      metrics:
        enabled:
          - dns
          - drop
          - tcp
          - flow
          - port-distribution
          - icmp
          - "httpV2:exemplars=true;labelsContext=source_namespace,source_workload,destination_namespace,destination_workload,traffic_direction"
      tls:
        auto: { enabled: true, method: cronJob }

A few choices worth borrowing:

• The UI has no ingress, on purpose. Hubble UI shows all your cluster's traffic, so you never expose it raw. An oauth2-proxy in front of Keycloak serves it at hubble.georgehomelab.com, the same single sign-on pattern every other UI here uses.
• TLS between the pieces uses Cilium's own CronJob, so kube-system doesn't depend on cert-manager.
• It's applied by hand, not by ArgoCD. kube-system is too foundational for a reconcile loop: one bad sync here can cut a node off the network. So it's a reviewed kubectl apply, and you watch the agents roll.

The three things it's actually good for

The live service map. Open the UI, pick a namespace, and the graph draws itself from real traffic. It's not a diagram someone drew; it's the actual edges, appearing as packets flow.

The Hubble UI filtered to one namespace (kubeflow), with per-edge protocols and flow counts.

Reading the network's decisions instead of guessing them. When a connection fails, the question is always "is this supposed to work, and if not, what blocked it?" Hubble answers directly:

hubble observe --namespace kubeflow --verdict DROPPED --last 50

The kernel already made the decision, and Hubble reports it as it happened: FORWARDED or DROPPED, and why. No more simulating NetworkPolicy rules in your head. That matters here: there are 48 Kubernetes NetworkPolicies and 6 Cilium policies on this cluster, and I can't hold all of them in my head.

For what it's worth, the cluster is quiet. When I swept all 20 nodes for drops, I found 24, all UNSUPPORTED_L3_PROTOCOL: non-IP frames Cilium simply doesn't route. Not one was a policy denial. That's a nice thing to be able to prove rather than assume.

DNS, where the policy asks for it. Cilium can see DNS queries and answers, but only for traffic passing through its DNS proxy. Here that means the Kubeflow profile namespaces, where a Cilium policy lets notebooks look up the package index. In those namespaces Hubble shows every lookup and its answer, which is exactly what you want when pip install hangs.

The thing I found when I went looking for HTTP

The draft of this article promised "which HTTP call 500'd". Then I went and checked.

Across all 20 nodes, Hubble had seen zero HTTP flows.

There are two reasons, and both are worth knowing before you enable the same httpV2 metric I did:

1. Cilium only parses HTTP when traffic goes through its layer-7 proxy, and it only does that for flows a policy tells it to inspect. None of my policies have HTTP rules, so Cilium never looks inside.
2. Most of my traffic is inside an Istio mesh. The sidecars encrypt pod-to-pod traffic with mTLS before it reaches Cilium's datapath, so even with an L7 rule, all Cilium would see is TLS.

So for the meshed workloads, which is most of this cluster, Hubble is a layer 3, layer 4 and DNS tool, and it's a very good one. HTTP behaviour comes from the mesh instead: Kiali and Istio's telemetry see every request, status code and retry, because the sidecars are the HTTP endpoints. That's not a failure of Hubble; it's the two tools working at different layers. But I'd have liked to know before I put HTTP in the article's subtitle.

Making it show HTTP

Zero flows is a fact about my policies, not about Hubble. To prove it, I gave Cilium one flow it was allowed to read.

The demo is two small workloads in a namespace that is deliberately outside the mesh: storefront calls orders every two seconds. Both are plain busybox: orders runs httpd on port 8080 with two small files to serve, and storefront loops over wget. The part that matters is the policy on orders:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: orders-http
  namespace: hubble-l7-demo
spec:
  endpointSelector:
    matchLabels:
      app: orders
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: storefront
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: /
              - method: GET
                path: /api/.*

Take away the rules.http block and this is an ordinary layer 4 rule: storefront may reach port 8080. With it, Cilium redirects that port through the Envoy proxy built into each agent, and the proxy reads every request. The policy now allows GET / and GET on anything under /api/, and nothing else.

The client sends four requests each round: two that should work, one for a file that doesn't exist, and a POST the policy doesn't allow.

hubble observe --namespace hubble-l7-demo --protocol http

storefront-556fcd89c7-gs8k9 -> orders-5b87b7c4bc-5bcq4:8080 http-request  FORWARDED (HTTP/1.1 GET http://orders:8080/api/orders.json)
storefront-556fcd89c7-gs8k9 <- orders-5b87b7c4bc-5bcq4:8080 http-response FORWARDED (HTTP/1.1 200 2ms (GET http://orders:8080/api/orders.json))
storefront-556fcd89c7-gs8k9 -> orders-5b87b7c4bc-5bcq4:8080 http-request  FORWARDED (HTTP/1.1 GET http://orders:8080/api/missing)
storefront-556fcd89c7-gs8k9 <- orders-5b87b7c4bc-5bcq4:8080 http-response FORWARDED (HTTP/1.1 404 4ms (GET http://orders:8080/api/missing))
storefront-556fcd89c7-gs8k9 -> orders-5b87b7c4bc-5bcq4:8080 http-request  DROPPED   (HTTP/1.1 POST http://orders:8080/api/orders.json)
storefront-556fcd89c7-gs8k9 <- orders-5b87b7c4bc-5bcq4:8080 http-response FORWARDED (HTTP/1.1 403 0ms (POST http://orders:8080/api/orders.json))

That's the real output, with the timestamps, namespace prefix and identity numbers trimmed so it fits. Three different stories in six lines:

• 200 in 2 ms. Allowed, served, and timed. The latency is measured by the proxy, per request.
• 404. The policy allowed it and the application said no. The network was fine; the file isn't there.
• 403, and DROPPED on the request. The policy refused it. Cilium's proxy answered with the 403 itself, in under a millisecond, and the request never reached orders.

That last pair is the one worth having. A 403 from your application and a 403 from the network look the same to the caller. Hubble shows which it was.

The Hubble UI on the hubble-l7-demo namespace: the storefront → orders edge, and the flows table with the HTTP method, path and the 200, 404 and 403 status codes.

The httpV2 metric I'd switched on months earlier finally had something to count, too. Straight from one agent's metrics port:

hubble_http_requests_total{destination_workload="orders",method="GET",source_workload="storefront",status="200"} 15
hubble_http_requests_total{destination_workload="orders",method="GET",source_workload="storefront",status="404"} 9
hubble_http_requests_total{destination_workload="orders",method="POST",source_workload="storefront",status="403"} 8

Request rate, error rate and a latency histogram per workload pair, with no sidecar and no change to the application. I read those from the agent directly, because nothing scrapes them yet (see the traps).

Before you copy this onto a real service, three things it changes:

• The policy is also a firewall. The moment a pod is selected by an ingress policy, everything that policy doesn't allow is refused. orders now accepts storefront on port 8080 and nothing else. Write an HTTP rule to get visibility and forget another caller, and you've caused an outage.
• The traffic takes a detour. Requests to that port now pass through a proxy on the node, so they pick up a little latency and depend on that proxy. It's one proxy per node, not one per pod, which is still far lighter than a sidecar.
• Flows now carry request details. The URL and headers are in the flow record, visible to anyone who can open the Hubble UI. Cilium can redact them, and I'd turn that on before pointing this at anything with tokens in a query string.

And it doesn't change the answer for the mesh. An HTTP rule on a meshed pod would still show Cilium a TLS stream. On this cluster, layer 7 through Hubble is for the workloads Istio doesn't cover.

How reliable is it?

Here's what the networking layer survives, and the evidence for each:

And the honest limits. The Relay and the UI are single replicas, so if the Relay is down you lose the cluster-wide view, though every node's agent keeps working. Each node keeps 4,095 flows in memory. A quiet node sees about 10 flows a second and a busy one close to 300, so that's anywhere from six minutes of history down to fifteen seconds, and about 45 seconds on a typical node. And nothing scrapes the metrics yet. All three are on the list at the end.

cilium status from one agent: Cluster health: 20/20 reachable, Hubble: Ok.

The traps that taught me

KubeArmor crash-looped my CNI

After I enabled Hubble, cilium-agent on two nodes went into CrashLoopBackOff with permission denied trying to create /sys/fs/bpf/tc. It couldn't pin its own eBPF maps. Nothing in the Cilium config was wrong.

The culprit was KubeArmor, the runtime security layer from the security episode. It generates a default-deny AppArmor profile for every pod, and that profile blocked the exact BPF calls Cilium needs. My CNI was being confined by my security tool: two layers of the stack fighting in the kernel.

The fix is one annotation, and the hard part is where it goes:

podAnnotations:
  container.apparmor.security.beta.kubernetes.io/cilium-agent: unconfined

It has to be top-level podAnnotations. daemon.podAnnotations looks right and isn't a real Cilium key, so it's silently ignored, and the agent gets confined again on the next roll. It also has to be this per-container annotation; a pod-level appArmorProfile: Unconfined isn't enough, because KubeArmor keys off the container one. Without it, the two nodes wouldn't recover even after a reboot.

That fix got undone more than once. An older copy of this manifest sitting on the control-plane disks kept putting the broken version back.

The Relay that crash-looped on two nodes

The Relay kept crash-looping, 105 restarts, with:

dial tcp <hubble-peer ClusterIP>:443: connect: connection refused

It only happened when the Relay landed on two particular nodes. On those two, Cilium's datapath answers the hubble-peer service with ICMP port-unreachable. The service is internalTrafficPolicy: Local, hard-coded in the chart, so the Relay can only reach the peer service through its own node, and on those two nodes that fails. On every other node it works.

I still don't know the root cause. I ruled out kube-proxy, socket load-balancing, Cilium's state database and the kernel. So the fix is a placement workaround, and I'm saying so plainly:

relay:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/hostname
                operator: NotIn
                values: [rke2-worker-dc3-app-a, rke2-worker-dc3-db-a]

Setting affinity replaces the chart's default, which includes a pod affinity that keeps the Relay next to a Cilium agent. Restate that too, or you'll break it differently. Since the workaround, the Relay has stayed on healthy nodes.

8 GiB of memory the scheduler couldn't see

The cilium-agent ran with no memory request at all, while actually using 217 to 460 MiB per node. Across 20 nodes that's about 8.3 GiB of real memory the scheduler didn't know existed. That's how nodes hit real memory pressure while still reporting headroom.

The fix is a request, and deliberately no limit:

resources:
  requests:
    cpu: 100m
    memory: 512Mi

512Mi sits just above the highest use I measured. There's no limit on purpose: a memory limit on the CNI agent is an off switch for the node's network. It made every node look about 512Mi fuller to the scheduler, and that's the point. The memory was already in use; now the scheduler knows.

The metrics nobody reads

All those metrics are switched on, and nothing scrapes them. There's no ServiceMonitor for Cilium or Hubble yet, so the drop and DNS rates exist and vanish. It costs almost nothing, but it's the first thing on the list.

Hubble or something else?

"Isn't this just what Kiali does?" is the fair question. The answer is no, and I run both. They work at different layers:

1. eBPF network observability, no sidecar: Cilium Hubble, Microsoft Retina, Inspektor Gadget.
2. Service-mesh observability, sidecar-based: Kiali with Istio, Linkerd Viz.
3. Packet capture: tcpdump, Wireshark, ksniff.
4. Commercial network monitoring: Datadog Network Monitoring, Kentik.

• Kiali and Istio win for per-request behaviour: status codes, retries, latency, mTLS. On this cluster that's where HTTP visibility comes from, because the sidecars are the HTTP endpoints.
• tcpdump is still the tool for "is this TLS handshake actually broken". It just doesn't scale past one flow.
• Retina is the pick if you're not on Cilium.
• Datadog wins on zero operations, if you'll send your flows to their cloud and pay per host.

Why Hubble here: it's eBPF, it knows Kubernetes identity, it's free, and Cilium was already my CNI, so it cost a few Helm values. For "what's talking to what, and did the network allow it", nothing else comes close for the price.

What's Next?

The invisible is now a live map behind a Keycloak login, and the network's decisions are one command away. What I'm doing next:

• Scrape the metrics: a ServiceMonitor for Cilium and Hubble, and a dashboard for drops and DNS errors.
• Keep more than a minute: export flows to OpenObserve, so "what dropped at 3 a.m." is a query, not luck.
• Run two Relays, so the cluster-wide view survives losing one.
• Find the real cause behind the two nodes that refuse the Relay, and drop the workaround.
• Let the SRE agents use it: an agent that reads DROPPED flows and proposes the NetworkPolicy fix.

If you're on Cilium and you've never switched Hubble on, you're sitting on an X-ray machine you haven't plugged in. Just check what it can see through before you promise anyone HTTP.

Follow me on LinkedIn

Author: George Ezejiofor