Last time, we introduced BGP as a protocol, but introduced the concept of autonomous systems (AS for short) and gave some examples. This time, we’ll dive deeper into the concept, which hopefully will give you a better understanding of the various types of ASes and how the Internet is organized on a global level.

But first, let’s start with how the numerical identifiers—the autonomous system numbers (ASNs)—work.

What are ASNs?

ASNs are how different ASes are distinguished in BGP. When two BGP peers connect, they each identify themselves with their own ASN. Furthermore, every route transmitted over BGP has an AS path attribute (see previous post) that describes which ASes the route goes through before reaching the destination.

In the original version of BGP, ASNs are 16-bit integers, which allows for 65536 distinct values. In 2012, this was deemed insufficient and a protocol extension was defined so that 32-bit ASNs became possible. Given that there are more than 100 000100\,000 ASes active on the Internet, this couldn’t have happened sooner.

Out of the original 16-bit ASN space, the following ASNs are reserved:

  • AS0: to identify routes that shouldn’t be announced through RPKI;
  • AS23456: to represent 32-bit ASNs in AS paths when talking to routers that don’t support 32-bit ASNs;
  • AS64496 to AS64511: for documentation and sample code;
  • AS64512 to AS65534: for private use; and
  • AS65535: for well-known BGP communities (we’ll get to this later).

From the additional 32-bit ASN space, the following ASNs are reserved:

  • AS65536 to AS65551: reserved for documentation and sample code;
  • AS4200000000 to AS4294967294: reserved for private use; and
  • AS4294967295: reserved for future, unspecified use.

Private ASNs exist to allow BGP communication between peers that don’t have a publicly allocated ASN for various purposes, which we’ll explore in the future. If the route is destined for the Internet, the private ASN must be stripped out of the AS path. To everyone else, the route would appear to come from some other ASN.

Note: in this series, we’ll only use ASNs reserved for documentation in examples.

How are ASNs assigned?

World map showing regions managed by each RIR

The world as managed by each RIR (source)

ASNs, like IP addresses, are considered Internet numbers, which are globally managed by the Internet Assigned Numbers Authority (or IANA). However, IANA doesn’t do mundane things like giving ASNs or IP addresses to people—instead, they delegate this task to the Regional Internet Registries (or RIRs), of which there are currently five.

The world is divided into five regions, each managed by an RIR:

  • African Network Information Centre (AfriNIC): manages the whole continent of Africa;
  • Asia Pacific Network Information Centre (APNIC): manages all of Asia east of Iran except the post-Soviet states and all of Oceania;
  • American Registry for Internet Numbers (ARIN): manages the US, Canada, some Caribbean islands, and apparently Antarctica (according to their website);
  • Latin America and Caribbean Network Information Centre (LACNIC): manages all of the continental Americas from Mexico southwards and the remainder of the Caribbean; and
  • Réseaux IP Européens Network Coordination Centre (RIPE NCC)1: manages Europe, Greenland, Western Asia (up to and including Iran), and all the former territories of the Soviet Union.

Typically, each RIR will only be willing to deal with you if you are either legally based in their region, such as being a resident or incorporated there; or have network presence, meaning at least a VPS, though some RIRs may require you to have a dedicated server or colocation space2. The exact policy is up to each RIR.

Some RIRs, such as ARIN, deal with end users directly, while others will only deal with you through a local Internet registry (LIR)—or if you become one yourself. In the latter case, the LIR applies for an ASN on your behalf and gives it to you3. In the APNIC and LACNIC region, there are also national Internet registries (NIRs) which can sit between RIRs and LIRs.

There are usually other requirements, such as the ownership of your own IP space, which can be requested from the same entities as ASNs, and at least two other ASes that are willing to peer with you. You’ll see why soon.

What relationships exist between ASes?

Now that we know what ASNs are, let’s dive into the sort of relationships that exist between ASes, from which we can work towards understanding the large-scale structure of the Internet.

We’ve already seen last time that ASes may provide transit to other ASes to help them reach other ASes to which they are not directly connected. Let’s consider the following diagram showing the relationship between a few ASes and the rest of the Internet. You can imagine this as a tiny part of the Internet:

A diagram showing the relationships between some example ASes

Here, solid lines represent transit and dotted lines represent peering without transit. This means that AS64502 can only reach AS64503 and not any other AS through their peering, but AS64502 can reach the rest of the Internet through AS64500. Among network operators, peering without transit is typically just called peering as opposed to transit.

From this, we can define several types of ASes:

  • AS64500 and AS64501 are transit or carrier ASes since they provide transit for AS64502, AS64503, and AS64504.
  • AS64502 and AS64504 are single-homed, meaning that they reach the remainder of the Internet (outside of peering) through transit provided by a single AS. Should its connection to that AS go down, or should it fail completely, then neither could reach the rest of the Internet. Since AS64502 peers with AS64503, it could still reach AS64503 through peering, but AS64504 has no such luxury.
  • AS64503 is multi-homed, meaning that it connects to the remainder of the Internet through transit provided by at least two distinct ASes. If one of AS64500 or AS64501 fails, then AS64503 can still reach the rest of the Internet through the other.
  • AS64504 is a stub AS, meaning it only has no peering and a single transit connection to the rest of the Internet, namely through AS64501.

Note that here, all the non-stub ASes have multiple connections, which means they have a choice in routing policy. For example, AS64502 may decide to send traffic to AS64503 directly through their peering instead of using AS64500 as transit, which is a shorter path and probably cheaper. Similarly, AS64501 could choose to reach AS64502 through the peering with AS64500 instead of going through the rest of the Internet. Typically, networks will prefer peering over going through transit, since sending traffic over a peering link has no marginal cost4 and it’s better to conserve the limited capacity of transit links to reach other destinations.

On the other hand, stub ASes don’t have a distinct routing policy from their parent AS, and as such are not actually autonomous. In such a case, AS64504 is effectively wasting an ASN, since all its prefixes could be announced from AS64501 and it wouldn’t make any difference to routing. If AS64504 is supposed to be an AS, this could only be a short-term situation while its operators try to peer with more ASes. Otherwise, the issuing RIR may demand the return of the ASN, especially if it’s one of the scarce 16-bit ones. This is also why having two ASes willing to peer is required for an ASN to be issued in the first place.

It is also useful to define a set of terms based on how close an AS is to the “rest of the Internet” based on transit relationships. Here, for example, since AS64500 provides transit for AS64502 and AS64503 to the rest of the Internet, AS64500 can be said to be upstream of AS64502 and AS64503. Conversely, AS64502 and AS64503 are downstream of AS64500. The downstream in a transit relationship is also referred to as a customer since they typically pay for the transit services offered by the upstream.

AS-sets, routing registries, and cones

At this point, it is useful to introduce the concept of AS-sets, which are simply a named set of ASNs. These are stored in Internet Routing Registries, such as:

  • AfriNIC, APNIC, ARIN, LACNIC, RIPE: each RIR operates their own IRR for their own users for no additional cost. This is often preferred by small networks and larger networks that don’t deal with many RIRs;
  • RADb: the Routing Assets Database, a public IRR that’s independent of any RIR, which costs $595/year at the time of writing to use;
  • AltDB: a free-to-use IRR that’s neither very trustworthy nor reliable.

There are typically two ways of naming an AS-set:

  • short names of format AS-NAME, such as AS-GOOGLE. Due to recent events5, new AS-sets of this form are no longer permitted in several IRRs such as RIPE;
  • hierarchical forms of the format ASx:as-name, ASx:as-name:as-name, …, such as AS16509:AS-AMAZON. Only the owner of the ASN can create AS-sets of this form (at least in properly authenticated IRRs).

AS-sets may contain ASNs and other AS-sets. A very common use of AS-sets is for defining the customer cone of an AS. This is commonly called the AS-macro or just “the AS-set” for an ASN. For ASxx, this as-set contains ASxx and all its downstreams. For downstreams that have downstreams, typically the downstreams AS-macro is nested inside the upstreams AS-macro. Let’s look at the previous example again:

A diagram showing the relationships between some example ASes

If the AS-macro for every AS in the diagram is of the format ASx:as-all, then:

  • AS64500:as-all contains AS64500, AS64502 and AS64503. Its upstreams would include AS64500:as-all in their AS-macros;
  • AS64501:as-all contains AS64501, AS64503 and AS64504. Its upstreams would include AS64501:as-all in their AS-macros;
  • AS64502:as-all contains just AS64502;
  • AS64503:as-all contains just AS64503; and
  • AS64504:as-all contains just AS64504.

Note that AS64503 is in the AS-macro and thus the customer cone of both AS64500 and AS64501 because it’s multi-homed. Also notice that the AS-macro of AS64500 and AS64501 are literally shaped like a cone in the diagram, which is why it’s called a customer cone. Note that the customer cone is commonly defined to also include the routes and IP prefixes of the downstream ASes instead of just the ASNs. These are what actually gets announced via BGP.

Now let’s look at the implications of the AS-macro in BGP sessions:

  • In a peering relationship, each peer announces their customer cone to the other peer. This means, for example, AS64501 would connect to AS64502 through the peering with AS64500 (assuming they do the sane thing of preferring peering over transit); and
  • In a transit relationship, the downstream announces their entire customer cone to the upstream, which in return announces routes to the whole Internet.

BGP views

I mentioned that upstreams announce routes to the whole Internet to their downstreams, but there are actually two ways this can happen:

  • Default route: announcing a route covering all possible IP addresses, i.e. 0.0.0.0/0 in IPv4 and ::/0 in IPv6. Since longer prefix lengths are preferred in routing tables, the default route has the lowest priority. Typically, single-homed networks without downstreams receive a default route from their upstream and regular routes from their peers, and this way, anything not reachable over peering will fall back to transit.
  • Full table: announcing every route on the Internet. This allows the downstream to intelligently pick the best upstream to use when multi-homed. Peering routes are thrown in as well, and the best route is chosen. A router with a full table will not have a default route, since any destination that can’t be reached by a more specific route simply doesn’t exist on the Internet. This option is typically used by multi-homed ASes, to take advantage of all upstreams, and carrier ASes, to be able to deliver full tables to their customers.

Upstream all the way?

We described upstreams and downstreams, but there’s one remaining question. Let’s go back to the example and consider what happens if AS64500 and AS64501 weren’t peered. Then, if AS64502 wants to reach AS64504, it would have to go through AS64500, the “rest of the Internet”, and AS64501 before aching the final destination:

Same diagram as before with peering between AS64500 and AS64501 severed

Does this mean AS64501 is an upstream of AS64500? Well, no. They would both be downstreams of some higher-level ASes. Yet, it’s clear that there are a finite number of networks, so at some point upstreams can’t have upstreams. However, when do we stop going upstream and start going back downstream in the AS path?

Enter Tier 1 networks. Tier 1 networks are large ASes that can reach the whole Internet through peering alone. This implies that the union of their own customer cone and that of all their peers is the entire Internet. Since such networks are not allowed to use transit without compromising their Tier 1 status, they all peer with each other6 to form a clique (in the graph theory sense7).

Since Tier 1 networks do not purchase transit, they have no upstreams. From the perspective of Tier 1s, there are three ways traffic on the Internet could be exchanged:

  1. Traffic flows through peering at a lower level and a Tier 1 is not involved at all;
  2. Traffic flows from one downstream of a Tier 1 to another downstream of the same Tier 1. In this case, the Tier 1 appears once in the AS path; and
  3. Traffic flows from the downstream of a Tier 1 through the Tier 1, then through peering to another Tier 1 to the destination. In this case, two Tier 1s appear in the AS path next to each other.

A corollary of this is that only a maximum of two Tier 1 ASNs can appear in any given AS path, and if there are two they must appear together. If this is not the case, then a route leak has occurred.

The following ASes are widely agreed to be Tier 1:

  • AS701: Verizon;
  • AS1299: Arelion (still often called by its former name Telia):
  • AS2914: NTT Communications;
  • AS3257: GTT Communications;
  • AS3320: Deutsche Telekom AG;
  • AS3356: Lumen Technologies;
  • AS3491: PCCW Global;
  • AS5511: Orange;
  • AS6453: Tata Communications;
  • AS6461: Zayo;
  • AS6762: Telecom Italia Sparkle;
  • AS6830: Liberty Global;
  • AS7018: AT&T;
  • AS7922: Comcast; and
  • AS12956: Telefónica/Telxius.

The following ASes are deemed by some to be Tier 1 but not all:

  • AS174: Cogent, which can’t reach AS6939 and its single-homed downstreams over IPv6 at all due to a peering dispute;
  • AS1239: Sprint, whose carrier services have been acquired by Cogent and is likely to be merged into AS174; and
  • AS6939: Hurricane Electric for IPv6, which can’t reach AS174 and its single-homed downstreams over IPv6 due to the aforementioned peering dispute. Note that HE is not considered a Tier 1 for IPv4 due to it purchasing transit from AS1299 to reach AS174, AS2914, and AS6453.

Internet Exchange Points (IXPs)

Tier 1 networks charge money to provide transit for other networks. Since they are an exclusive clique and only they can offer the ability to reach the entire Internet, they can charge a premium for it. In most cases, they will not peer with anyone else, requiring others to pay for transit. As a result, many smaller networks prefer to save money by peering with each other and exchanging as much traffic as possible over peering.

However, simple direct peering between ASes doesn’t scale as the number of connections required grows quadratically. For example, completely connecting two networks together requires a single link, but connecting 7 networks requires 21 links and connecting 100 networks requires an insane 4950 links.

Is there a better way? Well, remember network switches, which in the words I used last time, “connects all the devices attached to it together in such a way that all devices can talk directly to each other”? What if we connected a bunch of networks to a switch? This is an Internet Exchange Point or IXP.

Once the routers for two ASes are connected to the same IXP, they can communicate directly with each other and establish bilateral BGP sessions to exchange their customer cones. While this suits some networks that are particular about who they peer with, it’s still inconvenient for networks that wish to peer with as many networks as possible, especially on large IXPs with hundreds of potential peers.

For this reason, route servers were introduced. Effectively, these are servers that run BGP that don’t actually do any routing. Instead, it collects route announcements from all routers peered with it and announces all of them back to the peers. This means that everyone who is peered with the route server sees the routes of everyone else also peered with it. The route server doesn’t add itself to the AS path and keeps the original next hop attribute, so traffic is exchanged directly between the peers without going through the route server.

A typical IXP is allocated unique IPv4 and IPv6 prefixes to assign to each peer. These IPs are not routable over the public Internet, but are there to ensure peers can address each other. It is important for each IXP to have their own addresses so that the same network can join multiple IXPs. The route servers are allocated an ASN as one is necessary to establish BGP sessions.

Note that IXPs don’t necessarily have to be used for non-transit peering. Peers can exchange whatever routes they want through bilateral BGP sessions, including providing transit. For example, AS200351 (me) receives transit from AS6939 (Hurricane Electric) over ONIX.

Also note that while some IXPs, such as ONIX, are free to join, most IXPs charge some sort of membership fee. Typically, the bigger the IXP and the faster your connection to it, the more it costs. This is simply because of economics—faster connections cost more, and the larger the IXP, the more value they deliver to their users through the number of peers, especially valuable ones. As a general rule of thumb, content providers, such as popular websites and hosting providers, would like to peer with residential ISPs, where their users are. The IXP fees are usually worth it for networks that can shift a significant portion of their traffic from transit to peering.

How do I figure out how a particular network is configured?

It’s one thing to hear about the theory, it’s another thing to see how networks are configured in practice. Luckily, a lot of this information is available publicly. Here are several useful websites that will help you understand how any network is configured:

  • BGP.Tools, which displays information based on the aggregation of the full routing tables from many sources. You can use this to see the prefixes and the connectivity of any AS, for example, AS13335 Cloudflare or 8.8.8.0/24, Google’s public DNS resolver.
  • PeeringDB, where networks declare their routing policies and connectivity information, for example, Amazon. You’ll also find a large number of IXPs, such as ONIX.

I hope you’ve learned something about the Internet we all use every day from this. Next time, if I ever get around to it, I’ll talk a bit more about BGP route selection and tricks you can do to improve routing, and also anycasting.

Notes

  1. What a name… I always find it awkward mixing English and French like this. Also, everyone just says RIPE, even though that’s technically the forum and not the RIR. ↩

  2. For the uninitiated, this means renting a server rack (or part of one) inside some datacentre. ↩

  3. Since this is not a tutorial, I’ll not be recommending anyone. I may go into more detail if I ever do write one. If you want to get your own ASN today, you can join the IPv6 networking discord to talk to people who are into this stuff. ↩

  4. While it might cost money to get a peering link established, such as paying the datacentre to connect you with your peer, once the link is established, it’s effectively the same as having run an Ethernet cable—sending any amount of traffic costs the same, so you might as well send as much as possible. There is no increased cost on the margin by sending more. ↩

  5. Certain individuals thought it was funny to impersonate entities that they don’t control by creating AS-[COMPANY NAME], so certain IRRs banned this in favour of the other approach, which only the true owners of the ASN can use. ↩

  6. Well, several networks commonly referred to as “Tier 1”s engage in peering disputes with each other, in which they don’t peer and their non-multi-homed customers couldn’t reach each other. According to the strict definitions, this means they aren’t actually Tier 1 since they couldn’t reach the whole Internet, but they still act like it. The best example is the long-running dispute between Cogent and Hurricane Electric on IPv6. Despite the creation of this cake, this peering dispute has lasted for at least 14 years at the time of writing. ↩

  7. The dictionary definitions of clique in the non-technical sense all involve a small group that doesn’t welcome outsiders, which certainly describes the attitude of Tier 1 networks to peering. ↩