Traffic Shaping

I got asked a question recently, “How far apart could control plane nodes of a kubernetes cluster be and the cluster remain stable?”

The further apart they are geographically, the greater the network latency. How does network latency impact a cluster’s stability? Are there certain deployment architectures that work better when network latency is a factor?

How does network latency impact etcd’s ability to reach consensus through obtaining a quorum? How does it impact spinning up workloads? Upgrades? Key rotation? Workload transitions? High-Availability? Disaster recovery?

Network latency in a kubernetes cluster is probably something to think about.

What can you do to test this out practically?

  • I could have some servers sitting in a server room with hundreds of kilometers of ethernet cable. Not very practical.

  • I could spin up cloud VMs across the globe. But I’d lose fine grained control over the latencies, and I’d have to be worried about noisy neighbours, physical network latency, and other constraints that cloud compute gives me.

Instead, I decided the best way of doing research into this was to run a bunch of VMs locally on some dedicated big iron. That removed all the external variables, but I still had the problem of how to introduce network latency?

netem to the rescue!

Deployment Architectures

By spinning up kubernetes master nodes and worker nodes, and playing with the network latencies between VMs using netem you can simulate different deployment configurations and see how kubernetes performs.

For example, one simulation I tried was a two region deployment, with a master and worker in the same ‘region’ (i.e. low latency betwwen them), and then another ‘region’ with two masters and another worker. Between the two regions I introduced differing network latencies, starting at 10ms and working up to 100mS, to see how the cluster was affected. (I’ll leave it as an exercise for the reader to map this to real world availability zones).

Another simulation placed all the master nodes in one ‘region’ and all the worker nodes in another ‘region’, with varying latencies between them. Would that fare any better?

Evaluating Performance

But how do you determine the impact of network latency on cluster health?

  • Subjective analysis won’t do. Show me the data.
  • Neither will a binary “Yes, the cluster breaks” or “No, it seems to be fine”. Where are the limits?

To do this we need a repeatable test suite - fortunately kubernetes has end-to-end tests, and for the most part they are repeatable and deterministic, making them suitable for our purposes. We just need to look at test failures and weed out the noise.

The approach I took was to vary the network latency, run the end-to-end tests, record the failures, and where you see a huge jump in test failures you know you’ve reached the maximum usable network latency between nodes.

How to netem

Before we begin, it’s worth noting that the latency you add to an interface is unidirectional, meaning that you need to add it in both directions (assuming a symmetrical speed interface between nodes). So let’s get to it!

First of all, make sure to delete any existing traffic shaping:

# tc qdisc del dev {{ interface }} root

Now create the queue which we will then apply shapping to:

# tc qdisc add dev {{ interface }} root handle 1: prio

Add latencies to the classids:

# tc qdisc add dev {{ interface }} parent {{ classid-name }} handle {{ classid-prio }}: netem delay {{ classid-latency }}ms

And finally, apply traffic latencies for outgoing network traffic for each specified node and destination ip address:

# tc filter add dev {{ interface }} protocol ip parent 1:0 prio 3 u32 match ip dst {{ dest-ip }} flowid {{ classid-name }}

And of course you need to do this for all nodes in your test cluster.

To make sure it’s all taken effect as expected, you can use a simple script like print-latencies.sh on each of the hosts in the test cluster to make sure the network has been shaped as you are expecting.

Now it’s just a case of running the kubernetes e2e tests, record the failures, adjust all the latencies, and rinse, lather, repeat until you get all your tests data. From this you’ll be able to see what kind of deployment clusters with what network latencies between them are stable, and which ones are not.

Results

I’m not going to record the results of my investigation here because it’s very dependent on the software versions involved. It’s just not transferable beyond the bounds of my research. But I’d encourage you to see how latency impacts the software you’re deploying!

Limitations

One thing I didn’t look at was how network latency affects the installability of kubernetes - I started with a running cluster that I applied latency to.

It’d be a worthwhile exercise to see how installation was impacted as well - but deploying on VMs on big iron might be a little too far from a real-world scenario to give any meaningful results.