Dylan Hall, a REANNZ network engineer, explains some of the technical details that underpin rate-limiting.
In this post, he covers classification and enforcement, and how each are implemented on our Juniper-based network. This blog post expands on the rate-limiting information from last week.
There are two significant parts of the international commodity rate-limiting that need to be implemented in our Juniper routers. The first is classification, distinguishing one member’s traffic from another and determining how we want to treat that traffic. The second stage is enforcement, using the classification to divide up the available bandwidth fairly amongst those members trying to use the service.
First up, a little context and the diagram that Charles promised last week:
The diagram above shows our connectivity with AARNet and the points in the network where we’ll be applying the rate-limiting for commodity traffic.
Classification is achieved on Juniper routers using firewall filters. For those more familiar with Cisco hardware, these are roughly equivalent to an access list. We generate a firewall filter that lists all of our members address space and applies a three colour policer to each member (more on this in a moment). The following example shows the structure we’re using:
As each packet enters the network the classifier will identify which member the traffic belongs to and apply that member’s policer. It’s worth noting that the Juniper routers are able to implement these fairly lengthy classifiers in hardware so there is no performance penalty performing the classification. We also generate a similar firewall filter for IPv6 traffic. The IPv4 and IPv6 traffic share a policer so there is no advantage (or disadvantage) using either protocol.
The key to the classification stage is the three colour policer. As the name suggests, the three colour policer looks at the traffic and applies one of three labels to it: green, yellow, or red.
When configuring a three colour policer you must specify two rates, a Committed Information Rate (CIR) and a Peak Information Rate (PIR). The policer compares the traffic to these two rates (using a pair of token buckets) and determines if the average rate is below CIR (green), between CIR and PIR (yellow), or above PIR (red).
In the REANNZ context, green traffic represents a member’s regular entitlement to traffic. The yellow traffic is burst traffic that we allow if there is sufficient spare capacity. And lastly, the red traffic is discarded.
An example, consider a member with 100Mbps of commodity bandwidth who is able to burst up to 200Mbps. We would implement a policer with a CIR of 100Mbps and a PIR of 200Mbps. If the member’s average traffic was below 100Mbps all of their traffic will be classified as green. If they exceed 100Mbps, the first 100Mbps will be classified as green with the excess being yellow. For example, at 120Mbps the first 100Mbps would be green while the remaining 20Mbps would be yellow. If they exceed 200Mbps the first 100Mbps will be green, the second 100Mbps yellow, and the remaining traffic is discarded.
A more detailed explanation of three colour policers can be found in RFC2698.
For those familiar with Class of Service (CoS) and Quality of Service (QoS), it is common to implement multiple queues in hardware to provide a different level of service to different types of traffic, for example a low latency low loss environment for VoIP traffic. Unfortunately this approach isn’t appropriate for our commodity traffic as splitting member traffic across multiple queues based on it’s colour will lead to packet reordering.
Fortunately Juniper have addressed this with a feature where multiple colours of traffic can share a queue (therefore maintaining packet ordering) while treating green and yellow traffic differently.
A normal router will queue traffic for transmission as it arrives from other sources. When there is capacity to spare, the router will typically transmit the packets immediately, meaning the transmission queue is empty most of the time. When the interface becomes full, the router will continue to queue traffic but the number of packets waiting for transmission will grow as the queue becomes full and each packet will have to wait longer before it’s turn to be transmitted. Finally, if too many packets continue to arrive, the queue will become full and subsequent packets will be discarded (commonly referred to as tail dropping).
This behaviour can be observed in the wild fairly easily. An Internet connection with spare capacity will work well, delivering packets at a fairly constant rate. As the network becomes busy, packets will continue to be delivered but you’ll observe that some take longer than others to be delivered. Lastly, when the network becomes very busy, you observe most of your packets being delayed and some being discarded.
The feature we are using is called Packet Loss Profiles (PLP). PLP allows us to specify a different behaviour for each colour. Recall that each packet is classified prior to delivery into the queue and each member has their traffic colored independently of one another; one member may not be sending any packets while another is attempting to send too many packets. If a packet is red, it will never enter the queue. If a packet is yellow or green, it will enter the queue. Once in the queue, green packets will be delivered as expected. Yellow and green packets will accumulate in the queue awaiting transmission. When the accumulation exceeds 10% of the total queue capacity (an indication of link congestion), all of the excess yellow packets will be dropped from the queue. This formulation ensures that green packets will have minimal obstruction by queued yellow packets during delivery.
In practice this means that while we have capacity spare, the queue will typically be empty and all arriving packets (green and yellow) will be transmitted. When the sum of the green and yellow traffic exceeds the available capacity, the queue will begin to fill up and the excess yellow traffic will be discarded.
– – – –
We hope that this has given you a bit of insight into the way rate-limiting will be implemented from a technical point of view. Tune in next week when we explain how rate-limiting will be experienced from a member’s point of view as the network becomes busier.
As always, if you have any specific questions regarding your membership, please contact our engagement team.