❝Beating BGP is harder than we thought❞

▼ In a paper for the HotNets'19, seven researchers admit that "beating BGP is harder than we thought". (Discovered through Aaron '0x88cc' Glenn.) The researchers looked at techniques used by big content delivery networks, including Google, Microsoft and Facebook, to deliver content to users as quickly as possible. This varies from using DNS redirects to PoPs (points of presence) close to the user, using BGP anycast to route requests to a PoP closeby and keeping data within the CDN's network as long as possible ("late exit" or "colt potato" routing).

Turns out, all this extra effort only manages to beat BGP as deployed on the public internet a small fraction of the time. So it's probably not really worth the effort. Also interesting: when BGP is worse, that's usually consistent over relatively long timescales, and when things deteriorate over one path, they tend to also get worse over alternative paths.

That seems strange, as the authors observe that "BGP, the Internet’s inter-domain routing protocol, is oblivious to performance and performance changes." However, BGP isn't deployed in a vacuum. People either install capacity where BGP is going to use it, or they tune BGP parameters to use capacity that's installed.

So having automated traffic management doesn't seem to help much—or does it? Maybe all paths deteriorate together because the automated traffic management solutions do their job and distribute the traffic equally over the available paths, keeping performance the same.

However, there are some cases when one of the options (regular BGP, anycast, late exit, DNS redirect) performs a lot poorer than the alternative(s). So it's probably more important to focus on avoiding really bad paths rather than trying to pick really good ones.

Also interesting: apparently ISPs rarely include the EDNS client subnet option, which DNS redirect really needs to work well. However, the Google and OpenDNS/Cisco public DNS services seem to do support it (but not CloudFlare), so if you're experiencing poor performance towards a CDN, you may want to try using the Google or OpenDNS DNS servers.

Download the paper here.

Permalink - posted 2019-12-30

Bluesky: Twitter wants a federated Twitter

▼ Twitter's Jack Dorsey, in (of course) a Twitter thread:

Twitter is funding a small independent team of up to five open source architects, engineers, and designers to develop an open and decentralized standard for social media. The goal is for Twitter to ultimately be a client of this standard.

This is interesting on several levels. I'll mostly talk about the protocol design part of this, but before I do that: when has a business that's pretty much in a monopolist position ever voluntarily given up that position? They must really be feeling the pushback against "content and conversation that sparks controversy and outrage", and see this as a way out.

The monopoly aspect is even more noteworthy when considering what happened with instant messaging. Back in the day, we all used IRC and then ICQ. Then there were Yahoo, AIM and MSN (and a few more, I'm sure). At some point, the IETF came up with an open standard that allows everyone to run their own server, similar to how everyone can run their own mail or web server: Jabber or XMPP. But as quickly as networks merged and interconnected, new closed ones such as Whatsapp and Apple's iMessage. Unless I'm mistaken, iMessage, as well as Facetime, actually run on XMPP but don't interconnect with other networks. So we're still left with walled gardens owned by huge corporations.

Very relevant here is Metcalfe's Law:

a network's impact is the square of the number of nodes in the network.

In a network with ten people, there's (roughly) 100 possible conversations that can happen. In a network with 100 people, that's 10,000 conversations. With instant messaging, the network effect is very strong. Then again, most people predominately IM with people in their own country. The effect is that in each country, there's one clear winner in the IM space:

For "microblogging" such as Twitter it's even worse. For IM it's just about possible to use different apps to talk to different people, but the point about a service like Twitter is that everyone can react to everyone else. So you really need one unified network for it to work well.

So the challenge for the bluesky initiative is to come up with a way where everyone can run their own service independently, but they all talk to each other, similar to how you can run your own mail server and still mail everyone else who has email. (Although in practice, most people use Gmail or Outlook webmail.)

At first glance, this looks a lot like the existing XMPP protocol, that works exactly this way. However, XMPP solves a much simpler problem: one-to-one communication. In networking we call this unicast. But when you post something to Twitter, it doesn't just go to one recipient, but to all your followers—possibly millions of them. So what we have here is multicast communication: one sender, multiple receivers.

In principle, this is a relatively simple problem to solve. We have multicast routing protocols that work relatively well, so just bolt that logic onto XMPP and call it a day. However, the big problem will be to make it scale. On the one hand, you have people with millions of followers. That's a lot of work, but it's simple: Donald Trump's messages pretty much go anywhere. But what about the hundreds of millions of users who are followed by a handful of others and only post a message once a month? Just keeping track of all these relationships so the messages can flow in real time creates enormous overhead.

And then there's the issue of spam and unwanted content. How do you control those issues across domains running on servers that belong to different organizations with different use policies?

An interesting problem to think about, and I'm looking forward to seeing what bluesky comes up with.

Permalink - posted 2019-12-17

Time to turn off IPv4 (just a little)

▼ Another month, and we'll be living in the 2020s. And yet, 70% of the internet is still IPv4-only. (I'll be writing a story looking back on IPv6 progress the past decade in January.) So I thought: maybe I should draw a line in the sand and turn off IPv4 for my website. But then how will those 70% find me, and all the links to older content will be dead to much of the internet.

So instead, I decided to host some of the images on my old website muada.com and the new home of my blog illjitsch.com on an IPv6-only domain name. This means that browsers running on a system with only IPv4 connectivity will find that they can't load the image at the top of this page. There's no tricks involved: the browser encounters a cold hard error.

However, I didn't want to have a broken image show up on the page, because people might just think it's my HTML skills that are the problem, rather than them being stuck in the past IP-version wise. So the browser is instructed that when it can't load the image, it should load another. That has a nice big warning in neon colors. I also use an image map to let people click on the word "IPv6" to get to a page that explains what's going on. If javascript is turned off, there's just a line of text saying there is no IPv6 connectivity.

Note that browsers will cache images, so you may still see the correct skyline image even if you currently don't have IPv6 connectivity.

This is the HTML code. You can click on the images to try if they load manually.

<img src="http://images.iljitsch.com/images/skylinedenhaag2.jpg" width=1920 height=180 style="max-width: 1920px" alt="Image couldn't load - you don't seem to have IPv6 connectivity" onerror="this.onerror=null;this.src='/images/warn-img-no-ipv6.png';" usemap="#nov6link">

<map name="nov6link" id="nov6link">
  <area shape="rect" coords="425,110,500,150" title="Click here for an explanation" href="http://iljitsch.com/ipv6-warning.php" />
</map>

I hope that more people will do the same so people start to actually notice that they're behind the times by having just IPv4 connectivity.

Permalink - posted 2019-12-03 - 🇳🇱 Nederlandse versie