Securing the routing of New Zealand’s Research and Education sectors

Aaron Murrihy - Senior Network Engineer

REANNZ has been accepted as a participant of the MANRS initiative, becoming the first network operator in New Zealand to do so. Security best practice is at the forefront of what we do at REANNZ, internally through the protocols and systems that we implement and the partnerships we form. Now seems like the ideal time to share a post outlining the efforts that we’ve undertaken to ensure the security and stability of our membership’s routing on a global scale.

MANRS (Mutually Agreed Norms for Routing Security) is a global initiative that recommends actions that can be undertaken by network operators to improve the stability of the Internet and mitigate its most common routing threats. MANRS’ pragmatic and straightforward framework for implementing best practice routing security reflects and expands upon our own efforts in this area.

In order to achieve trust in global routing information, all participants must do their part to secure and ensure reliability of their little corners of the Internet.

Route Hijacking/Leaking

Route hijackers advertise address space belonging to an unsuspecting network in an attempt to draw that network’s traffic towards themselves. This type of attack can be used to impersonate services (banking, communications, etc.) as a man-in-the-middle (MITM) attack or can be used to simply disrupt connectivity to that network as a denial-of-service (DOS) attack.

Although malicious hijacking of traffic is of the utmost concern, as it can end in theft of funds or intellectual property, the majority of incidents tend to be accidental. Configuration typos and poor routing policy logic can lead to advertisement of bad routing information. These mistakes are incredibly easy to make, and if they’re being honest, many network engineers will admit to having made such blunders.

In order for a route hijack to be effective, an attacker must ensure their route is preferred over the real one. This can be as simple as “being closer” to most of the Internet. The number of networks (autonomous systems) on the path back to the originating network is one basic metric that is used to decide between multiple versions of the “same” route. The route with the shortest AS (Autonomous System) path is preferred.

Route hijacking by shortest AS path

There was an incident where an overseas university had their address space advertised to Google by another organisation. This university had recently swapped to Google Apps for staff and faculty, so as well as losing access to Google Search, YouTube, etc., all email was taken down. The university worked with its NREN to correct the issue, which took over 24 hours to resolve. It turns out the offending organisation had mistyped a configuration on their border router, advertising the incorrect route, and because they had a direct connection to Google, the false route was preferred.

The Internet uses more specific routes as the most basic metric for making routing decisions. This functionality is extremely useful for balancing traffic across multiple handovers or service providers. An attacker can split a route into multiple smaller routes and advertise them from anywhere. They will likely become preferred across most or all of the Internet.

Route hijacking with more specific routes

In 2018 a small Internet service provider in the US was making use of a “BGP optimiser” appliance. This appliance takes in BGP feeds from the Internet and then splits the Internet routes into more specific routes in order to control steering of traffic out of that network. However, due to poor routing policy these fake routes were accidentally leaked externally and attracted a chunk of the Internet’s traffic, overwhelming the network infrastructure in the area and taking a large number of services offline. Cloudflare have provided an excellent explanation and post-mortem of the event.

REANNZ has put a number of systems in place to protect the network and our membership from foul play in global routing and also to protect our membership and the wider Internet from REANNZ’ human error.

1. Exact route filters on member Internet handovers 

REANNZ actively maintains a close working relationship with the operators of our membership’ networks. This includes collecting an exact list of routes that each member expects to advertise. After being checked for legitimacy, this information is stored in an IP address management (IPAM) system. Route policy is then built from this information to ensure only the expected routes are accepted. This means should a mistake be made by a REANNZ member, it will not be seen by the rest of the world.

2. Limit number of routes that will be accepted across external peering connections 

If a peer network advertises a larger number of routes than the configured maximum, then the BGP session to that network will be immediately dropped. This is to protect against such events as the “BGP Optimiser” incident where a network, due to misconfiguration, suddenly begins to advertise a large number of erroneous routes. It is a very coarse protection and should be used with care as it can cause disruption on legitimate network behaviour, such as when Hurricane Electric, one of the largest network providers in the world, arrived on the Auckland peering exchanges adding their 108k routes to the existing 55k.

3. Additional route filters on external peering connections 

REANNZ adds route policy to external peering connections to ensure we can only export member routes that have originated from their Internet handovers. This is belts-and-braces protection against propagation of erroneous route information. It also ensures REANNZ cannot propagate routes between external peers which would in effect make REANNZ an accidental transit provider for those networks.

4. Automation of all Internet routing policy

The vast majority of REANNZ Internet routing policy is generated by automation (~37k lines of config). As well as making day-to-day operations simpler and quicker, automation reduces the substantial risk of human-error in our routing policy.

5. Management of Internet Routing Registry (IRR) objects for members

IRRs are a series of databases used to communicate routing intention. This intention can be used by peers, Internet service providers, and Internet exchanges to generate routing policy to ensure propagation of accurate routing information.

As incorrect IRR information can cause parts of the Internet to become inaccessible, REANNZ maintains IRR information on behalf of our members. All IRR objects are generated by automation using data from our IPAM system. We also have alerting in place to notify us when member IRR information is changed by another party.

6. Resource Public Key Infrastructure (RPKI)

RPKI is a technique for cryptographically matching a route to its origin network (AS). When importing routes from peers, REANNZ BGP routers can ensure routes have been originated by the appropriate network before accepting them. RPKI does have some failings and so is not a full solution to the problem of erroneous routing information, but it is a step in the right direction.

For more information on RPKI, REANNZ gave an operational presentation of our deployment at the Apricot Conference 2020 in Melbourne. You can watch it here on YouTube.

IP Address Spoofing

IP address spoofing is a technique where an attacker pretends to be another Internet user by forging their IP address when sending data. It can be used in conjunction with a route hijack to perform a man-in-the-middle attack. However, on its own this technique can be used in denial of service (DoS) attacks.

An attacker can send a request to a service on the Internet with a faked IP address, but the response will be sent back to the legitimate user. This behaviour can be further exploited by sending specially crafted requests to certain types of services (DNS, NTP, etc.) which solicit much larger (potentially >100x) responses. A single attacker sending 10Mb/s of requests, can generate a 1Gb/s of responses. 1,000 concurrent attackers (e.g. a botnet) can generate 1Tb/s of attack traffic against a single target! This is an amplification (or amplified reflection) distributed denial-of-service (DDoS) attack.

Amplification DoS attack

REANNZ’ membership occasionally see these types of attacks, and there are mitigations we can put into place while they’re occurring to keep the network and our membership functioning, but these are just stop-gaps. The only reliable method for stopping these types of attacks before they can occur is to ensure that packets with faked IP addresses are not forwarded onto the wider Internet in the first place.

Mitigations discarding 14Gb/s of DDoS traffic (red) destined for a member

7. ACLs on member Internet handovers

As a good citizen of the Internet, REANNZ places stateless access control lists (ACLs) on all of our member Internet handovers to limit IP addresses with which they’re allowed to send traffic. These ACLs are generated by automation (~21k lines of config) using the same IPAM database that is used to generate our route filters. Although, these measures will not protect our members’ networks specifically, it does serve to protect against reputational damage as they are unable to be used by nefarious parties for reflection-type attacks.

REANNZ is early in the process of enabling these ACLS on member handovers. This type of change has the potential to be very disruptive if not done carefully, so close collaboration with our members is of utmost importance. Expect to hear more from us on this in the future.

Conclusion

At REANNZ we take pride in our position as gatekeeper to the New Zealand’s research and tertiary education sectors and are committed to protecting the networks and reputations of our member institutions. That’s why we’ve invested significant time implementing and maintaining the capabilities outlined above.

Alone we’re only able to accomplish so much, but as participation of collaborative routing security initiatives, such as MANRS, increases, so does the security and reliability of the Internet as a whole. Therefore, REANNZ urges every citizen of the Internet to do their part.