Novatel Wireless pushed out a press release today to point out how their Ovation U727 USB modem was selected by PC Magazine as an "Editor's Choice" because it apparently "...provided the fastest average download/uplink speeds, beating out the latest USB modem offerings from competitors".
Each time we hear someone quoting PC Magazine about the speed of any EVDO device, it really bothers us. Why? Because the EVDO devices being tested are all built to specifications that qualify them as "EVDO Rev-A Capable", and that means they all have the same potential top speed.
What will affect the speed realized by the consumer is rarely ever going to be the device itself, and is more often than not a result of the device's signal to the cell tower; the load on that tower; the internet backhaul to that tower, or a combination of all or some of the above.
Ever since PC Mag started publishing their EVDO device reviews, we've read them with great interest. After their first few reviews, we hoped that they would detail their test methodology so we could poke holes in it. When it became obvious they weren't going to divulge the details we were looking for, we emailed the writers until we got someone to reply. Last January we exchanged several emails with Sascha Segan, a tech writer for PC Mag who has covered this technology for years. We must say, we respect his opinion.
Their latest reviews explain what we found out through private emails: what they do is take averages of tests that include public sites like speedtest.net. We questioned their logic -- explaining that to get "true speed results", the test environment would have to be closed off to public traffic that can use up some of the available bandwidth on a test server. Sascha replied (emphasis ours):
You’re right that for testing to be perfect, you need an Agilent 8960 and a Faraday cage, which only Consumer Reports has (and don’t think I haven’t thought about breaking in there in the dead of night to use their equipment!)
You’re right that the speeds I get with cards are dependent on network conditions, but that’s OK, in my view. I’m trying to come up with real world results, not theoretical results; I want to come up with numbers that would be accurate in the user experience
We also suggested that their testing could be slanted if one of the devices were defective or otherwise less than perfect, and Sascha replied:
Now, I could be really unlucky and just happen to have hit a situation where card A gets favorable network conditions in every test, but I’m doing my best to minimize that possibility.
Lastly, when we asked why they don't show the signal readings that each device had at the time that speed tests were recorded, and further pressed to get their opinion on signal in general, we got this:
I think you’d agree that the quality of the built in antenna(s) is a critical difference, if not THE critical difference in performance between these devices. Especially in low signal areas. ... I try to review things assuming that people will not have to buy extra accessories or parts. If you’re not interested in the role of built-in antenna performance, you’re probably going to come up with very different conclusions than I will!
In short, there was no argument that the details we brought up would in fact affect the outcome of testing to determine an "EVDO Speed King" -- they just simply choose to follow a different criteria for their 'real world' tests.
Our "real world" must not be the same as their real world. If we want to test maximum performance, our testing would at least have been done with each device operating under ideal signal conditions. In fact, we'd probably sit under a tower that we know has great backhaul so that our testing is most likely to judge only the device, and is less likely to be affected by signal related issues.
Speaking of tower load, its clear to us that where they were testing these devices, the towers were under a significant amount of load. The maximum bursts of over 1.5Mbps gives us a good idea that more than a single T-1 is serving as backhaul to the towers they tested from, but if all they could manage was an average of 650 ~ 800 Kbps, then there were probably a lot of users on those towers.
PC Mag contends that their testing represents "real world" results... and we can agree, but to use their methodolgy to judge maximum performance doesn't make complete sense to us. Its not like one device is going to be able to take any more that its fair share from a loaded tower... how can you possibly declare a 'speed king' when none of the devices had any chance of getting anywhere close to theoretical speeds? That's like taking a Corvette for a test drive in a Walmart parking lot on Black Friday. Good luck with that.
We've written our opinion on the subject of EVDO device speeds before, but it warrants repeating because so many people are going to quote the PCMag results:
Having sold thousands of these EVDO Rev-A devices to customers in all parts of the nation, its clear to us that signal to tower and load on that tower, are more likely to affect your "real world" speeds, than the choice you make about which device to buy.
Your choice for a device should be made based on features, not expected performance. You can expect all Rev-A devices to perform remarkably the same, given good signal to the same tower.
For those wondering if we simply don't like the Sprint U727... that really isn't the case. We don't think the U727 is a bad device. In fact, we sell quite a few of them, and don't have any issue at all with selling more of them. We just don't like anyone making claims that one device is a clear speed freak over the rest. It just isn't that clear to us.
Below is a compilation of PC Mag's "real world" EVDO speed testing. We feel that people should see the raw data without the commentary, and see just how close things really were.
Just for kicks, if you've never done a speedtest before, go do a dozen or so tests right now at SpeedTest.net and see how much variance you get across your tests. More often than not, tests from one minute to the next can easily vary by several hundred Kbps, and the difference in average download across the top five devices was only 85 Kbps.click on device to see fastest test results by EVDOforums members on EVDOmaps
|Sprint U727 USB Modem|| |
|Sprint EX720 ExpressCard|| |
|Sprint 597U USB Modem|| |
|Sprint 597E ExpressCard|| |
|Verizon UM150 USB Modem|| |
|Verizon KPC680 ExpressCard|| |
|Verizon 595U USB Modem|| |
|Verizon USB727 USB Modem|| |