Drama is brewing between mobile network operators in the US and third-party networking speed testing groups. On April 3, AT&T breathlessly announced that it had the fastest wireless network in the US with an average download speed of 40.7 Mbps, according to Ookla, which in previous reports had found T-Mobile to be the fastest.

AT&T’s press release also stated that “consumers overwhelmingly choose Ookla as the best way to check their network speed,” providing no evidence that sentiment toward speed testing providers was surveyed, nor explanation of how they determined that Ookla was used more than other speed testing services. Ookla owns Speedtest.net, which for naming reasons is likely to be near the top of the search results when looking for networking speed testing apps. The claim is not necessarily wrong, it’s just odd.

SEE: Special report: How 5G will transform business (free PDF) (TechRepublic)

A representative from Ookla told TechRepublic that the results were generated from “5 million consumer-initiated tests taken with Speedtest on over 1.5 million unique devices across all major carriers during the quarter,” though Ookla has yet to publish a separate editorial report on the US mobile market for Q1 2019.

Meanwhile, representatives from T-Mobile told Light Reading, in what editorial director Mike Dano called an “unsolicited email,” that “We think it’s important to recognize that these results are skewed by AT&T’s misleading 5G E marketing,” adding that “Naturally, seeing ‘5G E’ on their phones, AT&T customers were curious and many ran speed tests, giving AT&T disproportionately more tests in LTE Advanced areas.”

The “5G E” branding refers to AT&T’s deployment of LTE Advanced. It is not part of the 5G NR standard, and is not considered 5G by the industry standards body 3GPP.

OpenSignal ranked AT&T third just two weeks prior

On March 22, a report from OpenSignal found that of LTE Category 16 phones–capable of utilizing the features of LTE Advanced–Verizon’s network performed best on average, with a download speed of 29.9 Mbps. T-Mobile was in second at 29.4 Mbps, and AT&T in third at 28.8 Mbps. For standard LTE phones, T-Mobile and Verizon were tied at 19.4 Mbps, with AT&T at 18.2 Mbps.

OpenSignal’s report measures the speeds of all four US network operators as slower than Ookla’s, though the methodology is different between the two. OpenSignal uses an app that runs in the background and passively tests speeds, rather than relying on users to proactively test their speed.

Lacking a full report from Ookla, drawing meaningful conclusions to explain the discrepancy is somewhat challenging. Light Reading’s Dano casts doubt on the practice altogether, noting that “virtually every wireless network operator can point to data showing that it’s the best, fastest or most reliable,” and that a variety of factors can impact a user’s actual network speed, including congestion, differences between phones, user proximity and geographical topology, among others.

Subjectivity and demographics cloud measurement reliability

Phones that support LTE Advanced are capable of reaching faster speeds, so if the average speed of networks is measured with uneven numbers of LTE Advanced-capable phones, the results would be skewed. OpenSignal’s report separates LTE-A devices from others, though the publicly available numbers from Ookla cited by AT&T do not.

An Ookla representative told TechRepublic that the firm “did do additional analysis where we broke out devices capable of LTE-CA and found the outcome to be the same with AT&T having the fastest network in Q1.” (LTE Carrier Aggregation, or LTE-CA, is a feature of LTE Advanced.)

That said, AT&T does have a disproportionately high number of iPhone users, according to the testing volume in their database. Some 70% of tests of AT&T’s network were performed on iPhones, compared to 49% on T-Mobile, 54% on Sprint, and 62% for Verizon.

Because of OpenSignal’s reliance on background scanning–a function that Ookla contends is not supported on iOS devices–“there is no true way for OpenSignal to ever fully account for the actual performance of their network, because they can’t actually take measurements the bulk of devices that use the network,” according to the Ookla representative. The OpenSignal app is available on iOS, though the support of specific functionality could not be verified by TechRepublic.

Additionally, the Ookla representative highlighted the differences between their methodology and that of OpenSignal:

OpenSignal’s measurements are based primarily on background scans and the scans themselves are not actually a direct measurement of throughput. The speed measurement taken by a background scan, which is best used for measuring areas like coverage etc, is in no way as accurate as a dedicated Speedtest that is flooding a connection with data to assess what that connection can support at the time a person initiates a test. Given our incredibly large consumer-initiated test count during the period coupled with the close geographic proximity of our server network, our data set is an accurate reflection of the real-world performance consumers have on each carrier.

Additionally, OpenSignal is basing their assessment off a single month of data. We do not allow national level claims for a time period less than a quarter, because we find that longer periods of time control for any anomalies and show a leveled out view of typical carrier speed during the period.

While Ookla’s claims about sample time have merit, it’s not abundantly clear that “flooding a connection with data” is a useful measure of average, real-world performance (though it would make sense in determining the fastest speed). Likewise, the “close geographic proximity” would be closer to a best-case scenario, as not all services used by consumers have a geographical CDN.

Fundamentally, these two reports use different methodologies to attempt to ascertain what “real-world performance” is, though declaring supremacy from either and calling it the final word is never going to be accepted with universal consensus.

Learn more about the promises that (real) 5G offers. Check out “T-Mobile and Sprint promise low-cost 5G coverage for rural America, aiding remote workers,” and TechRepublic’s cheat sheet for 5G mobile networks and for 5G smartphones, as well as “Testing Verizon’s new 5G speeds exposed three major issues with the next-gen network” at CNET.