I never used to give search results a second thought as to validity. The responses I got seemed to fit what I asked for. That changed, when I was working on the article, "Why is my Internet different from your Internet?"
What I found
My first inkling that something was amiss came while running a simple test. I did a search on two different computers, expecting identical results. But, that did not happen. More digging led to my realization that search-result shaping is going on. Note to self — be more wary.
Then I read the New Scientist post, "US Internet providers hijacking users' search queries":
"Searches made by millions of Internet users are being hijacked and redirected by some Internet-service providers in the US."
Jim Giles, the author, explains:
"The hijacking seems to target searches for certain well-known brand names only. Users entering the term "apple" into their browser's search bar, for example, would normally get a page of results from their search engine of choice.
The ISPs involved in the scheme intercept such requests before they reach a search engine, however. They pass the search to an online marketing company, which directs the user straight to Apple's online retail website."
On the surface, this justifies my concern.
Is that legal?
In his article, Giles made mention that a class action law suit is already underway. I asked both Reese Richman and Milberg, the two New York law firms involved in the suit, for a comment. I did not receive one by post-time.
Six days after the original post, Giles added this update:
"Since the practice of redirecting users' searches was first exposed by New Scientist last week, we have learned that all the ISPs involved have now called a halt to the practice. They continue to intercept some queries - those from Bing and Yahoo - but are passing the searches on to the relevant search engine rather than redirecting them."
Okay, I get it. The legal stuff is complex and ongoing. None of which helps me right now.
Back to the research
Giles named Christian Kreibich and Nicholas Weaver as two of the research team involved in finding the redirection. I'll bet they can help. Bingo. I found a paper by them, plus Boris Nechaev and Vern Paxson called "Implications of Netalyzr's DNS Measurements."
Full of optimism, I read the paper. And, I was not disappointed by what they wrote:
"Target-dependent redirection: As we reported previously, Netalyzr identiﬁed multiple ISPs that use DNS to redirect web searches for popular sites, such as www.google.com, search.yahoo.com, and www.bing.com . Instead of visiting the intended search engines' IP addresses, the user winds up redirected to proxy servers. Some ISPs only manipulate Yahoo and Bing, while others manipulate all three."
The key word is identified. So there is a way to know if my search results are being tampered with.
I knew it was a long shot — being a holiday weekend - still, I tried contacting the members of the research team. Mr. Weaver, from ICSI at Berkeley, kindly responded, helping me figure out what's what.Kassner: To start, what is search-query redirection? Weaver: Search-query redirection is when the ISP redirects the user's search-engine requests through a proxy server which can then change how the query is processed or modify the user's search results. Kassner: OpenDNS, the DNS service I use, redirects search queries - if given permission - when non-existent domains and typos are found. Is that what you are referring to? Weaver: No we are not. Search-query redirection is a very different set of behavior from what you describe (the behavior you describe is "NXDOMAIN Wildcarding" or "DNS error monetization").
How search engine redirection works is that the user's computer asks the ISP DNS resolver for the address of the search engine, e.g., Bing. But the DNS resolver, instead of returning the valid answer, returns the address of a "proxy server".
This proxy server then receives all of the user's search requests and, depending on how it's programmed, may change the results. For example, on normal searches it may do nothing, but in this particular instance it would key in on searches issued from the browser's address or search bar.
Example, if the user typed in "CA" into Internet Explorer's search bar, the proxy would recognize this as being one of the keywords it was interested in and instead of returning the search results, the proxy would redirect the user through an affiliate program, so the user's browser ends up visiting the Computer Associates' web store rather than the search-results page.
In this process, the ISP and the company they work with would probably get paid by the final site through the affiliate program. We can't tell, but it seems plausible that both the affiliate program and the final site do not know how the traffic is directed to them.Kassner: The paper mentions Netalyzr as being a diagnostic tool. Was it developed specifically to deal with search-query redirection?
The major abnormality Netalyzr discovered refers to my having OpenDNS redirection enabled.
If they do find that their ISP is manipulating traffic in this way, they should both complain to the ISP and switch DNS resolvers.
Run the test
Follow this link to learn about Netalyzr and run your own test
I also wanted to pass along some advice from the EFF. If you run Firefox, there is an EFF extension called HTTPS Everywhere:
HTTPS Everywhere side-steps the problem by writing all requests in HTTPS.
One begins to wonder how much and what is going on — depending on your point of view - that's not written or in the fine print of EULAs and privacy policies.
I am grateful to the research team behind Netalyzr and Mr. Weaver for helping me understand what's going on.
Information is my field...Writing is my passion...Coupling the two is my mission.