When it comes to Internet Speed, it may seem like this is a cut-and-dry measurement of "how fast am I?" However, the deeper I dug into this, the murkier the waters became. I started out pretty happy - and let me warn you now, if you want to stay happy, don't read any more. Let me lay out for you my journey, and I will add clarity to how Internet speed is tested, and how it
should be tested - and what and why my favorite speed test web site is what it is.
The (Internet Super)Highway To Now
This is the road I took, in the post-dial-up modem world. I had Comcast as my cable TV and Internet provider, and this was tons better than the dial-up modems I had been using before. But, eventually (read: years later) I had reliability issues - reliability of the technology, that led to my problems with the reliability of their Technical Support technicians showing up at scheduled times. And the "I'm so sorry, I'll credit you $20 - oops, we already credited you $20 this month" didn't cut it when I had to call several times a month, and several days each month I had no Internet service.
So, I switched to AT&T. Now, with AT&T, things were great - for about 18 months. Speed was the same as Comcast, but customer service was awesome. Then, it broke down - something went wrong. The customer service remained great, but after a dozen (literally) on-site visits, fixing and replacing every piece of equipment inside the house and out to the main street drop - the speed just wasn't the mere 18 Mbps that I was paying for. And by that time, Comcast had zoomed past that performance - and UVerse had absolutely no public plans to boost their speed.
So, I did it. I split my communications providers - AT&T for TV, and Comcast for Internet. During this period, I relied heavily on 3 web sites that provide Internet Speed Testing: Ookla
speedtest.net,
SpeedOf.Me, and
TestMy.Net. I used these tests to help me determine when I was having issues, and how severe they were. During the UVerse time, I noticed that AT&T used Ookla's software (and presumably their infrastructure) to test Internet speed. And, the 3 speed sites gave me slightly different results, which I chalked up to different servers used for computing the speed. However, the results were similar enough at 18 Mbps, but I noticed a big difference at 50 Mbps.
The Speed Test Difference
So, to me, the way I would think to test, is to throw some data across, and time it - see how long it takes to send and receive. Further, I would try to "flood the pipe" - fill it with as much data as you can push, and see what its limits are. That seems simple enough, right? Ookla uses a Flash-based app embedded in your browser to test speed, so that didn't work from my iOS devices - but they do have an iOS app. Since I was concerned that Flash may skew results because of its overhead and lack of performance, I searched to find a non-Flash alternative, which led me to SpeedOf.Me. The great thing about these is the test on demand, but then the problem I had was intermittent speed slowdowns, which was hard to show if I had to go to a web page and click to start a test - over and over, all day and all night long. So, I searched to find a speed test that took samples over time, and came across TestMy.Net. The best of both words - non-Flash, and you can set up a repeated sampling every hour for as long as you want.
Once I switched back to Comcast for Internet, however, the disparity in testing between the 3 sites became apparent. On a 50 Mbps service, I was testing 58 Mbps, 60 Mbps with Ookla, 53 Mbps with SpeedOf.Me, and - get this - 39 Mbps with TestMy.net. Why such a huge discrepancy?
 |
| Ookla SpeedTest.net results |
 |
| SpeedOf.Me results |
 |
| Conglomeration of TestMy.Net results over time |
Now, we get to testing methodology. Remember my assumption that a speed test is simply dump some data, see how long it takes? Turns out that is not how Ookla does it. And, AT&T and Comcast (as well as many other ISPs) use Ookla as their testing software. Furthermore, when you search on the web to find places to test your speed, Ookla shows up at the top. So, here are the differences in their testing methodology:
- Ookla methodology uses what I would call a statistical hack. They do the speed test, divide the results into 20 "slices" - and discard the bottom 30% and the top 10% of speeds. Further, it "multistreams" the data - which means, it runs multiple, simultaneous parallel transfer streams to fully utilize your bandwidth. In addition, they have servers that are closer to you (mine was Troy, MI), so that it can test with fewer "interference" issues introduced by longer chains of connections.
- SpeedOf.Me methodology uses a multistream test as well, but it does not discard outliers. It uses a few servers (mine was Dallas) to more closely simulate what you would get in a real world scenario, but still uses Point of Presence (PoP) servers because, for example, you don't want an Internet test to show the speed between you and halfway around the world.
- TestMy.Net methodology by default uses a single stream, and does not discard outliers. If you want, you can change the test to Multistream. It uses a few servers (not hundreds like Ookla). The huge advantage here was, you can set up an automated test, and as long as your computer is on and your browser tab left open to the auto test page, you will get a test every hour for as long as you request, with the results stored in a database for your analysis.
Analysis of Methodologies
Statistical Outliers
Now, I don't know if you remember from your college Statistics class about discarding outliers, but there are times in which you discard outliers because they skew averages. Let me explain, that this is
not one of those times. When you are downloading files (say the latest OS update that is 4.2 GB), you really don't care what the trimmed average speed is going to be. You care how long it will take - for real. So the
real speed is the size of the file divided by the actual time it takes. In total. Without discarding the fastest and slowest portions. So in the case of testing Internet speed, discarding outliers is actually misleading.
Further, the skewed way in which they discard outliers is totally outright scandalous! Why discard the slowest 30%, and only the fastest 10%? It should be even, whatever it is. In discarding a larger portion of slower data points, you ended up with an inaccurate result that is skewed toward reporting a faster-than-reality speed.
In other words, OOKLA IS LYING TO YOU by fiddling with the results, to make it look faster than it is. And Ookla is the default speed test that all the cable companies use on their web site, and their techs will tell you to go to it to test the speed. Does that sound like nepotism?
Threading
In computing, "threading" is the method of taking these vast resources we have of computing power and network speed, for example, and splitting them so multiple things can happen "at once". I say at once in quotes, because in nanosecond timeframes, they are still happening only one at a time, but in our perception, it is so fast that it appears to happen in parallel. Threads seem to go faster, and thus better utilize the resources because the bottleneck is in the management of the data, not in the processing of the data, so you can push parallel "threads" at the "same" time and achieve an economy of scale. Indeed, web browsers and other Internet communication software makes use of these parallel threads to achieve better performance over the same connection.
So, if you use the Internet in a multithreaded way, it seems that the testing should show that usage, and how fast it is. If you were only to use a single thread (for example, that's how we used to do it in the old days of dial-up modems - a single phone connection, and a single data transfer going on over it), you would only effectively use a portion of the bandwidth. And indeed, this is what is shown by TestMy.Net's single threaded test - 40 Mbps vs. 53 Mbps. I argue, that tests should simulate actual usage - and, more importantly, should make use of all available bandwidth to tell me "what's the best possible scenario" for the line I have. So a Multithreaded test is the only way to accurately test your bandwidth.
Server Location
TestMy.Net argues that having servers located near you, on an Internet backbone, is not an accurate test of how we use the Internet, because when we go to a given web site (or other service endpoint), it could be located anywhere. True, I do understand this reasoning. However, I have to take issue with it, because when we run a network test, we don't want to see the actual bandwidth to an endpoint - we are already seeing that by going to that web site and downloading! No, we want to see, what is the total available bandwidth to the backbone, because when I am doing a speed test, I am trying to determine where the slowdown is. Is it in my house? Is it in the ISP? The Internet backbone? Or is it the remote endpoint server that I am connecting to? So, a speed test should tell me what's the fastest my connection is running, to the backbone, right now.
Data Collection Over Time
All things go in cycles - and a lot relies on the diurnal cycles of the Earth's rotation, including Internet traffic (tied to our sleep/wake cycles). However, when we have intermittent issues, it is difficult to trace down those causes without data taken over time, and we can see patterns if we collect data over time. Further, in the interest of troubleshooting, we can disconnect equipment to isolate issues, and run tests over time. However, it is critical that these data be collected at regular intervals. Perfect for a computer - and here is where TestMy.Net shines. Set up an Auto Test, 24 runs, and you have a 24 hour period of data collection. You can see the graph, download the graph, or download the raw data and do your own analysis.
Ubiquity of Ookla
After reading the above, you may have come to the conclusion that I think Ookla has us hoodwinked. You are correct! However, here is where it gets really insidious. To further skew what we, the public, find
out about our Internet speed - it seems that Ookla has garnered
contracts with all the major ISPs to provide their speed test results to
consumers. A Flash-based software, that discards outlying data points
in a skewed fashion, and uses questionable methods to report a speed skewed towards fast - I would call that misleading.
We all know how Google searches work - and indeed any search engine. Those who pay more money, get their results at the top. And Ookla shows up at the top of speed test searches. So even people who don't use their ISP's speed test, still use Ookla from the company's web site.
What Do They Have To Say To Me About This Article?
In writing this article, I solicited comments from the Tech Support departments of Ookla, TestMyNet, and SpeedOfMe. Ookla said they will not, under any circumstances, provide us users with raw test data (only the pruned data), and they politely refused to review this article and respond to these accusations of collusion and misleading. SpeedOfMe and TestMyNet were gracious enough to review my draft, and agree that I have a good grasp of the issues.
While I like SpeedOfMe because it is Flash-free, I find that hands-down, TestMyNet is the best overall speed test site. You can use it from any device, choose whether to use single- or multi-threaded testing, and schedule repeated testing over time. All 3 sites can store your data under a login so you can collect data over time.
What Do We Do Now?
Definitely, stop using Ookla for speed tests - they are lying to you anyhow. If you call your service technician, and they tell you to go to their page for speed test (e.g. http://www.att.com/speedtest or http://speedtest.comcast.net/), pay attention - you will see Ookla's logo in the graphics. Complain to them that Ookla is inaccurate and misleading, and that they should not use it for speed tests.
If you are so moved, write a letter of complaint to your ISP that they use Ookla, who skews the results to look faster than they are, and that they are lying to their customers. And, if you can think of anything else that concerned Netizens should do, post it in comments below.
