Networking

Search-query hijacks and redirection: What you need to know

Do you trust the results returned from your chosen search engine? Michael Kassner looks into search engine hijacking and redirection.

I never used to give search results a second thought as to validity. The responses I got seemed to fit what I asked for. That changed, when I was working on the article, "Why is my Internet different from your Internet?"

What I found

My first inkling that something was amiss came while running a simple test. I did a search on two different computers, expecting identical results. But, that did not happen. More digging led to my realization that search-result shaping is going on. Note to self — be more wary.

Then I read the New Scientist post, "US Internet providers hijacking users' search queries":

"Searches made by millions of Internet users are being hijacked and redirected by some Internet-service providers in the US."

Jim Giles, the author, explains:

"The hijacking seems to target searches for certain well-known brand names only. Users entering the term "apple" into their browser's search bar, for example, would normally get a page of results from their search engine of choice.

The ISPs involved in the scheme intercept such requests before they reach a search engine, however. They pass the search to an online marketing company, which directs the user straight to Apple's online retail website."

On the surface, this justifies my concern.

Is that legal?

In his article, Giles made mention that a class action law suit is already underway. I asked both Reese Richman and Milberg, the two New York law firms involved in the suit, for a comment. I did not receive one by post-time.

Six days after the original post, Giles added this update:

"Since the practice of redirecting users' searches was first exposed by New Scientist last week, we have learned that all the ISPs involved have now called a halt to the practice. They continue to intercept some queries - those from Bing and Yahoo - but are passing the searches on to the relevant search engine rather than redirecting them."

The Electronic Frontier Foundation (EFF) is also involved. They provided this technical analysis. Then a few weeks later, the EFF provided this update - correcting parts of their original analysis.

Okay, I get it. The legal stuff is complex and ongoing. None of which helps me right now.

Back to the research

Giles named Christian Kreibich and Nicholas Weaver as two of the research team involved in finding the redirection. I'll bet they can help. Bingo. I found a paper by them, plus Boris Nechaev and Vern Paxson called "Implications of Netalyzr's DNS Measurements."

Full of optimism, I read the paper. And, I was not disappointed by what they wrote:

"Target-dependent redirection: As we reported previously, Netalyzr identified multiple ISPs that use DNS to redirect web searches for popular sites, such as www.google.com, search.yahoo.com, and www.bing.com [18]. Instead of visiting the intended search engines' IP addresses, the user winds up redirected to proxy servers. Some ISPs only manipulate Yahoo and Bing, while others manipulate all three."

The key word is identified. So there is a way to know if my search results are being tampered with.

I knew it was a long shot — being a holiday weekend - still, I tried contacting the members of the research team. Mr. Weaver, from ICSI at Berkeley, kindly responded, helping me figure out what's what.

Kassner: To start, what is search-query redirection? Weaver: Search-query redirection is when the ISP redirects the user's search-engine requests through a proxy server which can then change how the query is processed or modify the user's search results. Kassner: OpenDNS, the DNS service I use, redirects search queries - if given permission - when non-existent domains and typos are found. Is that what you are referring to? Weaver: No we are not. Search-query redirection is a very different set of behavior from what you describe (the behavior you describe is "NXDOMAIN Wildcarding" or "DNS error monetization").

How search engine redirection works is that the user's computer asks the ISP DNS resolver for the address of the search engine, e.g., Bing. But the DNS resolver, instead of returning the valid answer, returns the address of a "proxy server".

This proxy server then receives all of the user's search requests and, depending on how it's programmed, may change the results. For example, on normal searches it may do nothing, but in this particular instance it would key in on searches issued from the browser's address or search bar.

Example, if the user typed in "CA" into Internet Explorer's search bar, the proxy would recognize this as being one of the keywords it was interested in and instead of returning the search results, the proxy would redirect the user through an affiliate program, so the user's browser ends up visiting the Computer Associates' web store rather than the search-results page.

In this process, the ISP and the company they work with would probably get paid by the final site through the affiliate program. We can't tell, but it seems plausible that both the affiliate program and the final site do not know how the traffic is directed to them.

Kassner: The paper mentions Netalyzr as being a diagnostic tool. Was it developed specifically to deal with search-query redirection?

Weaver: Netalyzr is a multi-purpose network measurement and debugging tool. One of the 80+ things it tests for is this search-engine redirection. Kassner: I ran Netalyzr on my network. I am hoping that you will guide me through the results. It appears that I have one major abnormality and several minor aberrations.

The major abnormality Netalyzr discovered refers to my having OpenDNS redirection enabled.

Weaver: There is a known bug in OpenDNS with how it handles IPv6 (the next generation Internet protocol) records, which we also detect. There are also some issues with your firewall or NAT in handling IP fragmentation (shown below).

Kassner: Of the tests that Netalyzr runs, which ones should we focus on if concerned about search-query redirection? Weaver: We specifically detect and flag the results as "Search engine redirection". Your report shows that this doesn't happen on your system.

Kassner: Are there any options — besides changing Internet service providers — if we suspect our ISP is using search-query redirection? Weaver: Use a third-party DNS service like Google Public DNS (8.8.8.8 and 8.8.4.4). Google public DNS does not do NXDOMAIN wildcarding or any other such redirection. Kassner: I realize there are lawsuits in play. Still, is there anything you can talk about that would help raise our awareness? Weaver: Most ISPs are not doing this, so most users are probably safe already, and if they are uncertain, they can use Netalyzr to detect this.

If they do find that their ISP is manipulating traffic in this way, they should both complain to the ISP and switch DNS resolvers.

Run the test

Follow this link to learn about Netalyzr and run your own test

I also wanted to pass along some advice from the EFF. If you run Firefox, there is an EFF extension called HTTPS Everywhere:

"We also still strongly recommend HTTPS Everywhere for users who don't want to have to read the fine print of their ISP privacy policy in order to ensure that they are actually talking to the sites they think they are."

HTTPS Everywhere side-steps the problem by writing all requests in HTTPS.

Final thoughts

One begins to wonder how much and what is going on — depending on your point of view - that's not written or in the fine print of EULAs and privacy policies.

I am grateful to the research team behind Netalyzr and Mr. Weaver for helping me understand what's going on.

About

Information is my field...Writing is my passion...Coupling the two is my mission.

Editor's Picks