Google

Real Life IT: The day the websites went out in Georgia

Google Safe Browsing warns users about malware-detected sites. Learn how one company dealt with its compromised website.

Credit: Pixel Insert

The holidays are always a good time to get caught up with family and friends. This past New Year's I was happy to run into a fellow system administrator who moved to Atlanta a few years back. My friend, whom I'll call Travis, had come back up north with his family to see relatives and experience some snow. Travis and I talked about the wonderful world of IT and he told me a tale about a malware problem on his company's website which we agreed was worth sharing here on the Google in the Enterprise Blog.

Travis is the sole IT guy for an organization in Georgia I'll call Glowstack.com. They have about 100 users running Windows with a mixture of Internet Explorer, Firefox, and Chrome browsers. Glowstack.com is their Active Directory domain and is the domain name used in all of their URLs, both internal and external.

Just a typical day in the IT trenches

"It was a busy morning, with a million things going on, as usual," Travis told me. "I got a call from a user who said 'Hey, what's up with our site? I'm getting a weird message in Chrome saying it contains malware and has been blocked.' My first reaction was that something had to be wrong with his browser - probably it updated itself overnight and had some freaky issue where it reported non-existent malware. I mean, the site was fine; I brought it up via Internet Explorer without any errors. I told him to try using Firefox, but he got a similar message. 'Something's probably wrong with your machine,' I said. 'Can you try reinstalling Chrome?'"

The user was willing to do that, and then Travis tried to access the site via Chrome and got the same message. (Figure A)

Figure A

Travis clicked "Advanced" to get more options. (Figure B)

Figure B

He found he could get to the site by clicking "Proceed at your own risk." Then he brought the site up in Firefox and beheld the following notification. (Figure C)

Figure C

Clicking "Ignore this warning" allowed access to the site.

Travis told the user to hold off on removing Chrome and explained how he could get to the site via either browser. This worked for the user, but Travis grimly knew the phone was going to start ringing again very soon.

Travis was able to get back to the warning screen above on a separate system then clicked the "Details about problems on this website" link. This produced the following page. (Figure D)

Figure D

Analysis needed

It was readily apparent that Google had flagged the company website in some kind of malware blacklist. The phone did indeed start ringing as it turned out that all internal URLs for Glowstack.com were affected by this problem - when viewed via Chrome or Firefox. Internet Explorer was not impacted. Travis drafted a hurried email to all users informing them of the issue and, for the first time since he could remember, advised them to use Internet Explorer for company websites until the root cause was determined and the problem solved.

That took care of the user base, but horror pervaded the organization once it was realized that the external URL for the company, www.glowstack.com, would present the same error to any customers or business associates who used Chrome or Firefox to access the page. Yes, ignoring the error and continuing to the site was possible, but that might not be readily apparent to non-tech-savvy customers and moreover the company's reputation was at stake. Would you buy a hot dog from a street cart vendor if a cop hovered nearby and said "I dunno, this guy seems shady, and I heard he might have a criminal record selling contaminated food?"

"I'll be honest and admit my first response to the alleged malware detection was to say a few unprintable things about Google and what I perceived as a bogus witch hunt," Travis said openly. "I didn't consider it remotely likely at first that the malware report had any validity. I guess it was the result of getting so many false positives from antivirus software - especially the crummy spyware that pretends to be antivirus software and gives you a list of 500+ threats present on your system then demands money to clean it - as well as dealing with melodramatic self-signed certificate errors. When you pretty much have to arm wrestle software into doing what it's supposed to do despite a bunch of Chicken Little prompts, you get a bit jaded about so-called threats. And that's exactly the danger here."

Travis called a quick meeting with department heads to fill them in on what was going on. They reviewed the advisory from Google in the screenshot above, took note of the references to Google WebMaster Tools and Google's WebMaster Help Center, and then started delving into the issue.

The "Uh-Oh" moment

The Google WebMaster Tools page provides a link titled "Malware and Spam." (Figure E)

Figure E

Travis and group investigated the section titled "Cross-site malware warnings" which explained what was happening: "When users visit a web page, browsers like Chrome check the content that's loaded to see if any part is potentially dangerous. When that happens, the browser will show a warning, alerting users that content from a site we've identified as being malicious is being loaded. In many cases, we'll also flag the original site as malicious, which alerts the webmaster and helps to protect potential users."

They accessed the "About malware and hacked sites" link, which advised taking the site offline, cleaning it to remove the malware, then requesting a malware review so Google could analyze the site and pronounce it fit for duty:

  • If your site is infected: How to clean up a hacked site.
  • Preventing malware infection: Best practices for avoiding infection in the future
  • Requesting a malware review: Once you're sure that all spam and malicious code has been removed, you can ask Google to review it. Google will check your site and, if it's now clean, will remove any warning label that appears in your site's listing on the search results page.

"We immediately debated whether it was better to leave the site up and potentially confuse customers, or take it offline and render it completely inaccessible (and also unavailable for review by Google). At the time I was still working with the angry assumption that Google had wrongfully blacklisted us, and my first response was 'We need to get them to take us off that list right away,' without considering the part about removing the malware," Travis told me. "Then a light bulb went on over my head and I said 'Uh-oh, what if the site really was hacked?'"

To complicate the issue, one of the directors came in and reported that some internal company URLs weren't accessible on a set of handheld terminals which use Firefox to complete certain web-based functions through internal websites. There was no input hardware to click on the option to proceed, so those terminals were dead in the water, which meant the business was suffering. As can be the case with directors, constant status updates were requested, so Travis wisely posted a liaison near his cube - with whom he kept up a verbal dialogue - to inform approaching employees what was going on and let Travis work uninterrupted. As Travis put it, "there's nothing less productive or more distracting than having to stop working on an emergency response to tell people over and over what's going on. It's like a fireman putting down the hose and reporting 'I've almost got that second story blaze put out now, chief - next I plan to get on the ladder and inspect the third floor.'"

Glowstack had an existing Google WebMaster Tools account, and Travis logged in and requested a review of the site so it could be checked again for malware. The site informed him a review would commence but this might take up to 24 hours. Unfortunately, though WebMaster Tools has a support page including a forum, it was apparent there was no way to get an actual human being on the phone for immediate assistance.

The time came to make the call and pull the plug on the web server. It was taken offline but left running so that it could be examined to find out if it truly had malware. The notion of restoring the site from backup was discussed. Tragically, the DAS unit (direct attached storage) which housed the past month of the company's disk-based backups had failed two days previously, taking the last full backup of the site with it. Full backups should have run immediately once the DAS was replaced, but a hardware error prevented these jobs from kicking off. The possible malware-compromised site they had just taken offline was their one functional copy.

"That was my bad, completely" Travis said. "The backup storage unit failed, the backup job didn't kick off and the alerts failed due to a misconfigured SMTP entry. I was too busy putting out user fires to detect the carbon monoxide that posed a much deadlier threat. You can't say this was a perfect storm, because a storm can't be prevented."

Travis arranged for customer service representatives to proactively reach out to frequent business customers where possible to inform them the site was temporarily unavailable (some of these customers interact with the site on a daily basis). He then engaged a professional IT security engineer to come to the facility and inspect the web server. Ultimately, malware was indeed detected on the system, which had been compromised through a lesser-known vulnerability. The vulnerability and the malware were both fixed, and then server was pronounced clean and restored to working order. It might sound like a simple process, but it took the rest of the day and much of the night.

Once the site was back users who accessed it still received the same warnings (this was deemed acceptable by company management, which decided any site was better than no site). Travis requested another review, and within 24 hours the site was pronounced clean. It was business as usual again - except for the post-mortem and steps taken to ensure this would be a one-off episode and not part of a repeat series.

So, what happened?

Google Safe Browsing was responsible for displaying the malware warnings to users. Wikipedia states this service "provides lists of URLs for web resources that contain malware or phishing content. Google Chrome and Mozilla Firefox web browsers use the lists from the Google Safe Browsing service for checking pages against potential threats." Internet Explorer wasn't impacted by this problem since it uses a separate mechanism (SmartScreen filter) to notify users of malware-affected sites. The site malware was picked up by Google Safe Browsing, and then Chrome/Firefox users were presented with warnings.

This incident is similar to what recently happened to Netseer. On February 4th, cm.netseer.com, an advertising site, was labeled as having malware by Google.

Companies don't need to possess Google WebMaster Tools accounts for Google Safe Browsing scans to be conducted against their sites (however, WebMaster Tools can help you address problems found); all sites are rated regardless of whether they are owned by Google customers or not.

What do I do if Google reports malware on my site?

The first and foremost thing you should do is address the malware issue; Google Safe Browsing will not pronounce your site clean until it truly is malware-free. Google provides a diagnostic page you can use to get more information about malware detected on your site:

http://www.google.com/safebrowsing/diagnostic?site=mysite.com

(replace mysite.com with your actual site URL)

You will see a page something like Figure F.

Figure F

This is similar to the screen Travis viewed for his company's website when it was stricken with malware. In the example above, useful clues are shown regarding the location of the malware and what the malware is doing.

The Google WebMaster Tools "Cleaning Your Site" page recommends the following process:

Scan your computer using an up-to-date scanning program to identify any malicious code the hackers might have added. Be sure to scan all your content, not just text-based files, as malicious content can often be embedded in images.

If your site has been infected with malware, check the Malware page in Webmaster Tools. (On the site dashboard, click Health and then click Malware.) This page lists sample URLs from your site that have been identified as containing malicious code. Sometimes hackers will add new URLs to your site for their nefarious purposes (for example, phishing).
  • Use the URL Removal tool in Webmaster Tools to request removal of hacked pages or URLs. This will prevent the hacked pages from being served to users.
  • Report phishing pages to the Google Safe Browsing team.
  • Use the Fetch as Google tool in Webmaster Tools to detect malware that might be hidden from the users' browsers, but served to Google's search engine crawler.
  • Review the antiphishing.org recommendations on dealing with hacked sites (PDF).
  • If you have other sites, check to see if these have also been hacked.

If you have access to your server, follow these additional steps:

  • Check to see if any open redirects in your site have been exploited.
  • Check your .htaccess file (Apache) or other access control mechanisms, depending on your website platform, for any malicious changes.
  • Check your server logs to see when files were hacked (bearing in mind that hackers can alter your logs). Look for suspicious activity such as failed sign-in attempts, command history (especially as root), or unknown user accounts.
Once you've solved the malware problem (and changed all related passwords!), you should use WebMaster Tools to request a review of your site so as to facilitate getting your site off the malware list. (Figure G)

Figure G

Don't have a Google WebMaster Tools account? Use this link to get started and add your site using these steps (the cost is free and there are many other useful options besides those geared towards dealing with malware). You'll have to verify ownership of the site ownership using a customized html file you need to upload to your site. It's a good idea to set this account up before your site runs afoul of malware; Google states via a blog post from last June that "signing up with Webmaster Tools helps us communicate directly with webmasters when we find something on their site, and our ongoing partnership with StopBadware.org helps webmasters who can't sign up or need additional help."

What have we learned today?

The morals of the story here are multifold:

  • Run anti-malware software and ensure it can send alerts to the appropriate personnel.
  • Make sure you review security issues for software and services you administer. Sign up for newsletters and alerts so you'll stay ahead of the curve.
  • Always patch all vulnerabilities, or mitigate risks using relevant security advisories if patches aren't presently available.
  • Conduct periodic security scans of your sites where possible, to detect any issues before they adversely impact the organization. Executives who object on financial grounds should be reminded that cleanup will quite likely be costlier than prevention.
  • Take any and all reported malware threats seriously.
  • Any problem with your backup environment (or redundancy capabilities) moves your business that much closer to the edge of a cliff. That's the kitchen fire that needs immediate extinguishing, rather than the small ashtray fires making up so much of day-to-day IT life.
  • Think about whether you need to put critical sites on the same domain. In the case of Glowstack.com, their internal URLs were affected by this problem - not a big deal for desktop or laptop users, but those handheld terminals were down, something which would have been avoided if they'd pointed to a non-Glowstack.com URL such as deviceglowstack.com (for internal sites setting up an alternate domain would require minimal effort involving DNS and certificates). As a result, employees had to scramble to make up for the loss of productivity.
  • Realize there is no 100% guaranteed checklist to cover all bases; you have to operate not from a reactive standpoint but from a perspective that can visualize all the components and working parts to assess potential pain points or "the next rivet to pop." This is more of a mindset than a concrete step.

Where can I get more information?

Here is a FAQ for Malware and hacked sites. Google has a blog page with helpful updates, as well as a forum discussing malware and hacked sites. Google WebMaster Tools presents a page on preventing malware infections, additionally.

Finally, Google provides a Safe Browsing API which developers can use to "permit applications to check URLs against Google's constantly updated lists of suspected phishing and malware pages."

Also read:

About

Scott Matteson is a senior systems administrator and freelance technical writer who also performs consulting work for small organizations. He resides in the Greater Boston area with his wife and three children.

7 comments
kjfaucheux
kjfaucheux

We had a perfect storm where I worked. Only thing was, it was caused by crappy management and I got hammered for it. We always ran "hot and close to the edge". Interesting read though on how things can go from bad to dire really quickly.

Paul A Thomas
Paul A Thomas

First off I agree with BFilmFan. Internal and external namespaces should never be the same! The next point is that Travis left the site up on line and compromized the security of visitors and everyone in the company visiting the site. What would happen if the malware spread through the network or affected a client's network? Malware warnings in a browser or pop-up anti-virus warning should be explored first by looking for the source, not wondering what browsers are showing the warnings and which ones aren't. Worry about Google later. If malware does exist either as an SQL injection or compromised .htaccess file etc. these need to be expored and cleaned up first. When that is done Google can be notified. One of the main reasons malware is dropped on servers is that the web software is out of date, files have insecure permissions and admins don't understand TCP/IP network security in a web environment. Personally I wouldn't have posted this article if it happened to one of my friends, save his embarrassment. Well I might embarass a Systems Admin friend - if he reads this ... He asked me to edit his DNS zone to include a new A record. Next day he emails me and says the domain isn't resolving and if I could check it for him. I ask what he is doing ... he says "I'm in the process of setting up an IIS web server ... I'm almost ready to launch" >_

mist27
mist27

Just curious, not being in IT specialist,from what you say, I assume you can have an internal AD and an External AD, If so How do the two link ? If this takes too much input to answer Maybe you could point me in a direction that will help thanks you

skyeenter
skyeenter

Been a committed Firefox user for years. But Google has been getting sh**ty with them lately so some features were becoming problems. Started using Chrome, becoming comfortable and started trusting it features. One of the things I always liked about Firefox was the "Email Link" from the file menu that was included in the broweser. I needed to load an app for Chrome to have that feature. Found an app and started the download. That's when my night mare began. It was a casual download, just like Firefox. And then I noticed more crap ware coming along with the download. Tried to hit cancel and close the download window. Inaccessible! Opened Task Manager fast as possible and killed the process, which killed all open applications on my system. Did the restart of Firefox and Chrome only to find a new search browser U Search.net. Started Malawarebytes search, which did not find any threads. . Started Microsoft Security Essentials deep drive and left for the day for it to run through the night. Returned the next morning and MSE was locked about 1/4 way through the search. Killed the search. Clicked on Start in Win 7 to search programs. It opened an at home legal program I had on the HD. Clicking on law program to close it brings up start menu. Click on start menu it brings up program. Now I know I'm hacked big time. Immediately shut down system and reboot into Safe Mode. More night mares. The laptop has a biometric fingerprint reader. Safe Mode loads and opens to Administrator login screen. No biometric reader available, the code was written in the boot log to start after the text code. One problem! No text password! Only one user account with no password. Completely locked out of machine. Insert CD to boot bios from CD drive. Win 7 Safe Mode says unable to change bios setting while in Safe Mode. CD drive boot inoperable so can't use HP Recovery Disc or Hiren's CD Boot Disk, or reinstall WIN 7 OEM from CD drive. Try F8 repair settings, Command Prompt, System Restore etc, all reboot back to Admin Log On Screen, which requires Admin text password on machine that only has biometric password. Finally pressing and restarting all the F keys I find in F11 I HP help reset/restore screen opens. Out of several choices I select Command Prompt which opens DOS screen. Open Regedit. Start searching for registry changes that will delete passwords, safe mode, allow boot sequences changes and what ever else I need to get back to the desktop so I can start removing the malware. Call HP and MS help desks for their help. I ultimately realized that the Level 1 support in both of these organizations is useless. But none of them will relinquish the case file and send it to a higher level. After I got into Command Prompt and Regedit I had to walk the "kid" at MS through how to enter command code. WTF, I'm suppose to be paying for that? So now I'm working on my own doing more searches for help. It's better than arguing with undereducated tech help. And I haven't even covered the call quality of the international connections. All I can say is a big thank you to everyone from Google Chrome developers who leave the gigantic holes in their code, the app creator who allowed the tag along code to install, and the f**king hackers who ingeniously and maliciously have been my constant companions for the last three and one half business days. And MS for changing their login feature in Win 7 so there's no bypassing the login like in XP. In addition, I want to thank the Corporate Thieves at Microsoft and HP for their hijacking attemps at my finances to help me with this problem. Their extortion is no different than the hackers demanding ransom ware purchase. But then they're corporations and exempt from that comparison. I have had to cancel appointments, turn down walk in business, process credit cards on paper charge slips and simultaneously manage three computers doing fixes and searches. My last resort is pull the hard drive, install in a HD caddy, save some of my program files and blow up the HD, if I can't eliminate the malware.

BFilmFan
BFilmFan

If I say it once as an AD Architect, I will say it a 100,000 times. Your internal AD namespace should NEVER EVER be the same as your external.

Mark W. Kaelin
Mark W. Kaelin

Malware compromised websites are a major headache for everyone. What steps have you taken recently to combat this problem?