Glasnost: Results from tests for BitTorrent traffic shaping

Home Glasnost Tests Create your own test Results Contact

Over the last three years, a large number of users from locations around the world have used our tool, Glasnost, to test whether their BitTorrent traffic is being shaped by ISPs. On this page, we present results from the tests conducted between January 1st, 2011 and November 5th, 2011.

The source code of our tool has been publicly accessible for more than 3 years now. You are welcome to download and inspect the code. Please contact us if you find any bugs or have questions, comments, or suggestions.

We published a technical paper describing our data collection and analysis methodology in the 7th Usenix Symposium on Networked Systems Design and Implementation (NSDI 2010). The paper explains in detail the potential sources of measurement errors and the checks we apply to minimize their impact on our results. You can download the paper in pdf format here.

Links to older releases of Glasnost results can be found here: February 1st, 2009, November 9th, 2008, and July 25th, 2008.

1. How accurately can users detect BitTorrent traffic shaping by ISPs?

We designed Glasnost to enable individual users to detect if their BitTorrent traffic is being shaped by ISPs. We can control the accuracy with which users can detect traffic shaping. But, in the process, we have to make some hard trade-offs between false positives, where the test indicates traffic shaping by ISPs even when they are not, and false negatives, where the test fails to detect traffic shaping even when ISPs are shaping traffic.

For details on how we can choose the trade-offs, please follow this link.

In brief, we can set the controls to detect any traffic shaping that reduces BitTorrent bandwidths by more than 20% with a 4% chance of false positives. Alternately, we could configure the controls to detect any traffic shaping that reduces BitTorrent bandwidths by more than 50% (higher false negatives) with less than 1% chance of false positives (lower false positives).

2. What can we infer about traffic shaping policies of ISPs?

Glasnost has been designed to enable individual users to detect if their traffic is being shaped by ISPs. Glasnost data from individual user tests can be used to infer if an ISP is deploying traffic shaping for at least some of its customers — by checking if the percentage of Glasnost tests for an ISP that indicate traffic shaping far exceeds the chance of false positives. However, Glasnost data cannot be used to accurately infer how widely traffic shaping is deployed within an ISP, i.e., the percentage of the ISP's users who are affected by traffic shaping.

For more details on how we can infer shaping policies of ISPs, please follow this link.

3. Are ISPs shaping BitTorrent traffic?

We present our analysis results for BitTorrent traffic shaping by ISPs world-wide in the table below. The table shows, for each ISP, the number of Glasnost tests ran by the ISP's customers as well the percentage of tests for which we detected traffic shaping of (a) BitTorrent uploads on any random port, (b) any TCP uploads on BitTorrent ports, (c) BitTorrent downloads on any random port, and (d) any TCP downloads on BitTorrent ports.

Note 1: When trying to infer whether an ISP is shaping traffic, it is important to compare the percentage of tests that detected traffic shaping with the chance of false positives. The greater the difference between the percentage of shaping detected tests for an ISP and the false positive rate, the more likely it is that the ISP is deploying shaping for at least some of its customers.

Note 2: To obtain the results, we configured our analysis such that there are few false positives (estimated to be less than 1%), but we fail to detect any traffic shaping that affects BitTorrent bandwidths by less than 50%.

Note 3: Our estimate of false positives, derived in our NSDI 2010 paper, is based on measurements conducted over wired (cable / DSL) access networks. The false positive rate might be higher for wireless access networks like WiMax or Cellular broadband networks. For such networks, it is better to allow for a higher false postive rate when inferring the presence of traffic shaping.

Note 4: We aggregated the results for different autonomous systems (ASs) owned by an ISP. For example, Tiscali UK and OPALTELECOM are two different ASs owned by TalkTalk. So we merged the results for user tests from both ASs under TalkTalk.

4. Where can we download the raw data and analysis scripts?

You can download the raw data collected during Glasnost tests from Measurement Lab. Please follow the instructions on the page.

The tools that we used to analyze the data can be downloaded from here.

We encourage you to analyze the data and inspect our scripts. If you have any comments, questions, or suggestions, we welcome you to contact us.

5. Inferring shaping policies using data cited in New York Times article on Glasnost

Glasnost project was featured in a New York Times article on Novermber 14th, 2011. The article cited the percentage of Glasnost tests for which we detected traffic shaping for ISPs world-wide. When inferring whether an ISP is deploying traffic shaping, it is important to view those percentages in the context of the potential measurement errors, all of which were not emphasized sufficiently in the article. In particular, we wish to emphasize the following:

Note 1: The results cited in the article were obtained using an analysis configuration that detects any traffic shaping that reduces bandwidth by more than 20% (compared to 50% in the table above). With this more aggressive configuration, the chance of false positives are higher, around 4% (compared to 1% in the table above).

Note 2: More importantly, the article cited aggregated results from 4 different BitTorrent tests (BitTorrent uploads, BitTorrent downloads, TCP uploads on BitTorrent ports, and TCP downloads on BitTorrent ports), each of which has an measurement error of around 4%. That is, a user is reported to have observed traffic shaping, even if any one of the four BitTorrent tests she runs detects shaping. So the resulting measurement error in the aggregated results could be as high as 4 x 4% = 16%.

Note 3: When mapping AS names to ISPs, the analysis used an older (2008) version of the mapping table from IANA.

You can download an xls file containing the results cited by the New York Times article here.

As the aggregated results (column H in the .xls file) have a high chance (16%) of potential false positives, it is hard to infer with high confidence that any ISP for which the percentage of users that detected shaping is smaller than 32-48% (a factor of 2 to 3 larger than false positive rate) is actually shaping traffic. A more accurate inference can be drawn from the results of individual BitTorrent tests (columns C through G in the .xls file) which have a significantly lower chance of measurement error at 4%. To further improve the accuracy of inference, we recommend using results presented in the table above, where the measurement error is estimated to be lower than 1%.

For example, for US ISPs like AT&T and Verizon the percentages of tests that were reported to have detected traffic shaping fall within the range of false positives. Results presented in the table above (with tighter bounds on false positives) shows more clearly that there is no evidence of traffic shaping by these US ISPs.

On the other hand, the article correctly reports that BitTorrent traffic shaping appears to be more common in certain European countries like Britain. The percentage of tests that detected traffic shaping are so high (70% or higher) for some European ISPs that they exceed the chance of being accounted for by false positives (16%) alone. In fact, results presented in the table above (with tighter bounds on false positives) also show that some European ISPs are likely deploying BitTorrent traffic shaping.

Contact

This page is part of the research on residential broadband networks at the Max Planck Institute for Software Systems.
The results presented on this page were compiled by Marcel Dischinger and Krishna P. Gummadi on 28. November 2011 as part of the Glasnost project.

If you have questions or feedback, you can contact us via e-mail: