Testing for suspect traffic shaping / throttling

[Seán]

Every mobile network mast has an uplink to provide connectivity to that mast in much the same way as any telephone exchange.  Many rural masts get their connectivity over a microwave wireless link to another mast.  Masts in areas with good fibre infrastructure may get their connectivity over fibre and some also provide microwave connectivity to the rural masts. 

While a microwave link can provide plenty of capacity for the nodes of a single mast, some masts may be fed off a hierarchy or mesh of other masts with microwave links.  This can lead to severe congestion upstream such as with the main fibre connection if it does not have enough capacity.  This can lead to even quiet masts experiencing poor throughput. 

Some ISPs (particularly mobile data) appear to use traffic shaping in many areas where BBR enabled hosts such as Content Delivery Networks (CDN) get priority over other types of traffic.  This is particularly noticeable in rural areas that face high network contention.  One good example of this is where general web browsing and downloading is incredibly slow while at the same time YouTube plays Full HD (1080p) on some videos without a glitch. This is due to Google using the BBR congestion control with its hosts (which Google developed), whereas most hosts use another TCP congestion control such as CUBIC.

In the past, Three did traffic shaping by port #.  This led to the popular Ookla Speed test delivering 20Mbps+ test results when many websites were taking 10 times longer to open pages than on a 5Mbps DSL connection.  Back then it was also possible to exploit this traffic prioritisation by making a VPN connection over port 8008, which Ookla uses.  Three no longer prioritises by port # from what I can tell.   


Testing for suspect traffic shaping

As hosts using the TCP BBR congestion control appear to escape the suspect traffic shaping, testing simply involves downloading a test file from a known fast host and then from a host that appears to use TCP BBR.   If the ISP does not have traffic shaping and there is no peering issues along the route to the host, then both files should download at roughly the same speed.


Test #1 - Measurement Labs ndt5 (TCP CUBIC) vs ndt7 (TCP BBR)

Measurement Labs runs its speed test over a single connection with the test server. This is different to speedtest.net, fast.com, etc. that make lots of simultaneous connections.

The Google Internet Speed Test uses M-Lab's speed test ndt7, whereas the Measurement Lab website uses ndt5 at this time of posting. From my testing, both use the same test server, so the main difference in speed is due to the TCP congestion control algorithm, i.e. BBR vs CUBIC. Ideally, both should return the same test results.

• Google: https://www.google.com/search?q=internet+speed+test
  
• M-Lab: https://speed.measurementlab.net/#/
  

Test #2 – Heatnet vs Fosshub

Heanet’s servers are fed by multiple 10Gbps links, so any congestion is very unlikely at that end. As they are based in Dublin, all peering should occur within Ireland.  Fosshub on the other hand uses Cloudflare as its CDN, which uses the BBR congestion control. 

Compare the download speed of Kodi 64-bit from Heanet and Fosshub:

• Heanet: https://ftp.heanet.ie/pub/xbmc/releases/windows/win64/
  
• Fosshub: https://www.fosshub.com/Kodi.html
  

Test #3 – Leaseweb vs Vultr

The Leaseweb hosting provider has test files up to 10GB in the Netherlands (when testing from Ireland), but the host serving its test files do not appear to be TCP BBR enabled.  The Vultr hosting provider also has test files, which appear to be BBR enabled. 

Time how long it takes to download the 100MB file from the the Leaseweb and either Vultr host:

• Leaseweb (NL & DE): https://kb.leaseweb.com/network/link-speeds
  
• Vultr NL: https://ams-nl-ping.vultr.com/
  
• Vultr DE: https://fra-de-ping.vultr.com/
  

If there is no noticeable difference between the Leaseweb and Vultr NL speed, try comparing the two DE test files.


Test #4 – TestMy London vs Colorado servers (Requires a TestMy account)

With most Irish ISPs the London server generally performs much better than the Colorado Springs server. This makes sense as Colorado Springs is in the centre of the United States. However, this TestMy host is using TCP BBR, whereas the others use another TCP congestion control algorithm. (Previously the Los Angeles server in Canada appeared to be BBR enabled, but no longer is)

TestMy simulates an actual file download by timing how long it takes to download a known block size into the browser's cache, making it a useful test on phones where it's difficult to time a file download. Don't confuse it with the popular Ookla or Fast speed tests which use very different test methodologies to determine what the connection's speed currently gets up to.

Note: Someone pointed out to me that the Colorado Springs server is only accessible while logged into TestMy. Otherwise it tests against Dallas, which is not BBR enabled.

These two links are set up to immediately start a 25MB download test against the London and Colorado Springs (Log in before testing):

• London: https://uk.testmy.net/b/dl-25MB
  
• Colorado Springs: https://co.testmy.net/b/dl-25MB
  

The following are test result examples on Three on the 4G+ mast I pick up from home – London vs Colorado Springs:



The Three 4G+ mast I pick up here very rarely seldom gets over 40Mbps on the London server even early in the morning, yet the Colorado Springs server often gets over 60Mbps even at peak time!

The Three mast in Donegal town (Abbey Hotel) does not appear to be affected. With line of sight of the mast, I regularly get over 100Mbps, sometimes approaching 180Mbps with the London server:



Update 25 Aug '20: I moved the TestMy test down as the Colorado Springs test server is currently only available to members logged in. I added a second Vultr test link as Three does not always prioritise the Dutch test links.

Update 3rd December 2020: I have since revised this post as it's clearly not the ISPs prioritising specific hosts or IP ranges that I thought previously, but instead influenced by the choice of the TCP congestion control algorithm.

[Imogen]

Well, that's confirmed a few things! We are on a coastal rural three band 20 mast.
Test 1 London 13.6 Mb, Los Angeles 18.4
Heanet 40 s v Fosshub 22 s
Leaseweb 62s v Vultr 49s

Actual speedtest on speedof.me 10.8 DL, 18.9 UL and quite slow ms today, 64 ms but we usually get 45 ms. This is mid afternoon before the evening congestion.
The sad thing is I actually think 10 Mb is "quite good!" these days. If only it would actually stay there...

By the way we found a cheaper source for those Yagi aerials, but doesn't look like getting one would really help much.
https://www.korexmall.com/lowcostmobile-...612638.htm

[Seán]

That's a great price for that pair of band 20 antennas, half that of Amazon! 

At that price, I think it's worth upgrading, even just to improve the prioritised traffic.  Three prioritises most major CDN traffic, so a few main bandwidth hogs such as Windows updates, popular YouTube videos and Google Drive syncing would benefit.  Whenever Three upgrades the backhaul serving your area (e.g. preparing for 5G), you should get the full benefit then. 

While travelling home with someone from Donegal, I tested London vs Los Angeles in Donegal, Dunkineely and Killybegs around 8pm with TestMy.  This shows how severe their traffic shaping can get in affected areas, London vs Los Angeles:

Donegal town (5pm):


Dunkineely (7:38pm):


Killybegs (7:55pm):


While going through Dunkineely, I had to manually choose a smaller 12MB block as it was taking too long, whereas the Los Angeles test flew with the 50MB block.  From looking at the "Test in Progress" graph, it would likely cause buffering issues with bandwidth sensitive services such as live streaming, video calls, etc:

   

It was only within the past year that Donegal town got upgraded as it had similar traffic shaping, unlike now where the nearer London test runs quicker as I would expect.

[ShaunB]

Hey Sean,

Just wondering can a VPN help avoid this traffic shaping with 3? Currently unable to leave my router on 4G as the speeds in the evening are terrible

[Seán]

If you can find a VPN server on a host that Three prioritises, I reckon it would get around the traffic shapping. Sadly, I have tried many VPN servers, but so far had no luck finding one that didn't lasted more than a short period.

Around two years ago, port 8080 had priority over regular HTTP traffic. As Ookla's speed test runs over port 8080, it meant its speed tests run much higher than what one could achieve normally. It also made it relatively easy to exploit - Just make a VPN connection over port 8080 and it would run as fast as the Ookla speed tests. Eventually Three closed that loophole, more likely from Ookla noticing something suspicious.

Later on when Three started priotising various web services such as YouTube/Google traffic, I found the odd host that happened to run a VPN server (e.g. from HMA). It was just a matter of testing various server locations and running speed tests and once a test flew, I kept the VPN connection established for the rest of the day. That trick only lasted a short period and lately I haven't found a VPN server that got prioritised in at least a year. The only brief exception was the Cloudflare WARP+ as Three does prioritise Cloudflare traffic. However, even that is no longer prioritised, likely from Three working out which hosts to exclude.

[Seán]

I have now found a VPN that gets past Three's traffic shaping, Hide Me, which I'll post about in a separate thread.

For those with Android or iOS, the CloudFlare 1.1.1.1 App gets around traffic shaping to an extent. I originally thought it no longer worked, but it appears to be capped around 20Mbps. This is still more than adequate for streaming or downloading something that's much slower otherwise.

[Seán]

I have since been informed by Measurement Labs that the huge variations in speed I am experiencing is likely due to the TCP congestion control. This became clear when Google's speed test switched to ndt7, which switched from TCP CUBIC to TCP BBR.

From further testing, this clearly looks to be the case as I get very different speeds depending on whether a web server is configured with TCP BBR or another congestion control such as TCP CUBIC. I have revised the first post as it's clearly not the ISPs prioritising specific hosts or IP ranges that I thought previously.