Network engineering firm identifies flaws in comScore broadband speed tests

Cable ISPs: new broadband test makes our service look slow [ArsTechnica]

Broadband speed tests conducted by research firm comScore, cited by the FCC in its National Broadband Plan, aren't always accurate according to Netforcast, a network engineering consulting firm. The FCC cited comScore's broadband speed tests to justify a statement that actual broadband speeds are only 40-50% of speeds advertised by service providers.

comScore's results are derived from a group of survey participants. Each participant has a comScore speed testing application installed on their computer which takes random samples of broadband speeds by downloading a file ranging from 1 MB to 15 MB. The application then calculates the size of the file divided by the time it took to download to determine the broadband speed.

But Netforcast has uncovered several errors in comScore's testing methodology that create lower broadband speed results. Those errors include the fact that most broadband connections are shared via wi-fi, with many systems and devices within a home accessing a broadband connection simultaneously. If one of those systems is using the broadband connection at the same time as comScore's testing, a lower speed would result due to the sharing of the broadband connection. comScore's tests do not compensate for this variable. 

The comScore testing also "severely limits the accuracy of its results" because it only uses a single TCP connection for each test. Netforcast contends that most speed testing services use multiple TCP connections. 

comScore's test results may also be flawed due to a fundamental miscalculation of the size of a megabyte of data. Netforcast believes that comScore may be calculating a megabyte at one million bytes of data rather than actual size of 1,048,576 bytes per megabyte. Such an error would result in a miscalculation of broadband speeds by 4.5 percent in every case.

Comments

[Paul Templeton]

The only thing I agree with here is possibly the TCP connections.
Most users know if multiple devices are using their connection and will usually disconnect other devices before doing the test.
As for the way the bits are calculated, that's kinda like the pot calling the kettle black.
Internet providers like to measure in Megabits while the rest of the world thinks in Megabytes. Because of this most people think they are getting more than they are by a factor of 10.
Quit crying about testing procedures and give us the speeds the rest of the civilized world gets. Not just download speeds but also upload speeds.
Bring the capacity of your networks up to par and maybe you will quit treating those of us who use the allotted bandwidth like criminals.