Recently, we sent out an email letting our members know that we will be implementing rate-limiting on our international commodity Internet network. We received a nice email back asking for further information about what information we will provide to the membership about their network usage and how rate-limiting will be implemented. Below is a slightly modified version of the email response we sent.
The below is a general explanation of rate-limiting. On Friday 27 March we will publish a more technical explanation of how rate-limiting is being implemented (I have been told that there will be diagrams).
We have had an overwhelming response to our new international commodity Internet due to the quality and low cost of the services available. While we expected to eventually need to implement rate-limiting, we thought we would have more time to test multiple implementations and experiment.
Regarding member network reporting, we are working on a real-time, member portal; unfortunately, it is not yet available. We are estimating that the member portal will be available in six months. In the interim, we will be sending PDFs of the type we have shared with many members in the past. These PDFs provide each member with information describing: usage of national and international capacity, comparative ranking of usage (other members anonymised), usage of caches, etc. We have built a system that will send the PDFs on a monthly basis to the members.
Answering the policing mechanism is going to take a bunch of text. You have been warned.
Firstly, we should make sure you know that domestic and R&E traffic is not affected by the rate limits. Rate-limiting will be applied only to international, commodity traffic. Thus, the rate-limiting will be applied at the edge of our network as traffic heads to and from SYD and LAX.
Overview of rate-limiting
With that out of the way, REANNZ currently provides 2 Gb/s of bi-directional capacity to Sydney (SYD) and Los Angeles (LAX). The inbound links are now at their limits, with SYD occasionally crossing over the 2 Gb/s mark. To address this, we will apply rate-limiting in the following manner.
1. Each member will have their traffic assessed based on their allocated and purchased capacity, hereafter their “total capacity”.
2. REANNZ will ensure that members have access to their total capacity across the SYD and LAX connections. REANNZ sees the combined capacity purchased by all of our members as a resource that can be shared, but we implement a set of rules to make sure that members have access to their purchased capacity and that other members do not unfairly take advantage of the community purchase.
3. When a member begins to send traffic at volumes greater than their total capacity, REANNZ will begin to tag the traffic as “drop eligible”. The decision to drop traffic is not implemented at this point in time.
4. If the SYD or LAX link has spare capacity, members will be allowed to burst over their total capacity. We will monitor bursting to encourage members to purchase their correct capacity and not utilise the capacity purchased by the other members.
5. If the SYD or LAX link is at capacity, members who are sending traffic in excess of their total capacity (thus, sending packets marked drop eligible) will have packets dropped. The packets marked drop eligible will be dropped randomly by the router. REANNZ will NOT allow members burst capability to push the SYD or LAX links over capacity.
6. REANNZ will evolve and improve the way that rate limiting is applied based on how the network behaves and from feedback from our membership.
The above steps cover how rate-limiting will be implemented. There are some additional details that may be of use to you.
East-west traffic split
REANNZ’s two connections carry traffic from different directions. Currently, the traffic balance is split unevenly between the two directions. SYD handles about 60% of the total traffic; LAX handles about 40% of the total traffic. There is slightly more traffic moving over the SYD link than the LAX link. In aggregate, each member’s total capacity is thus split between the two links.
Rate-limiting is implemented using a formula that takes into account the SYD and LAX links. The member’s total capacity is expected to be split across both links– specifically, a member does NOT have access to their total capacity over EACH link.
Given our knowledge of the overall traffic volume, we will allow each member access to 70% of their total traffic to be applied to the SYD and LAX links. This means that the member has access to 140% of their purchased capacity at any given time.
When there is capacity available, we will allow members to burst up to 125% of their total capacity across each of the links. This means that when the capacity is available, the member has access to 250% of their purchased capacity. We expect members who are consistently bursting beyond their total capacity to purchase additional capacity.
Hopefully the above helps to explain and answer some of your questions. We are preparing a full engineering response to be published on our blog next week.
If you are in Queenstown for eRNZ, see you there. Otherwise, feel free for any of your people to track me down with questions. And as always, feel free to send me questions.