The holidays are always a good time to get caught up with family and friends. This past New Year's I was happy to run into a fellow system administrator who moved to Atlanta a few years back. My friend, whom I'll call Travis, had come back up north with his family to see relatives and experience some snow. Travis and I talked about the wonderful world of IT and he told me a tale about a malware problem on his company's website which we agreed was worth sharing here on the Google in the Enterprise Blog.
Travis is the sole IT guy for an organization in Georgia I'll call Glowstack.com. They have about 100 users running Windows with a mixture of Internet Explorer, Firefox, and Chrome browsers. Glowstack.com is their Active Directory domain and is the domain name used in all of their URLs, both internal and external.
Just a typical day in the IT trenches
"It was a busy morning, with a million things going on, as usual," Travis told me. "I got a call from a user who said 'Hey, what's up with our site? I'm getting a weird message in Chrome saying it contains malware and has been blocked.' My first reaction was that something had to be wrong with his browser - probably it updated itself overnight and had some freaky issue where it reported non-existent malware. I mean, the site was fine; I brought it up via Internet Explorer without any errors. I told him to try using Firefox, but he got a similar message. 'Something's probably wrong with your machine,' I said. 'Can you try reinstalling Chrome?'"The user was willing to do that, and then Travis tried to access the site via Chrome and got the same message. (Figure A)
Clicking "Ignore this warning" allowed access to the site.
Travis told the user to hold off on removing Chrome and explained how he could get to the site via either browser. This worked for the user, but Travis grimly knew the phone was going to start ringing again very soon.
It was readily apparent that Google had flagged the company website in some kind of malware blacklist. The phone did indeed start ringing as it turned out that all internal URLs for Glowstack.com were affected by this problem - when viewed via Chrome or Firefox. Internet Explorer was not impacted. Travis drafted a hurried email to all users informing them of the issue and, for the first time since he could remember, advised them to use Internet Explorer for company websites until the root cause was determined and the problem solved.
That took care of the user base, but horror pervaded the organization once it was realized that the external URL for the company, www.glowstack.com, would present the same error to any customers or business associates who used Chrome or Firefox to access the page. Yes, ignoring the error and continuing to the site was possible, but that might not be readily apparent to non-tech-savvy customers and moreover the company's reputation was at stake. Would you buy a hot dog from a street cart vendor if a cop hovered nearby and said "I dunno, this guy seems shady, and I heard he might have a criminal record selling contaminated food?"
"I'll be honest and admit my first response to the alleged malware detection was to say a few unprintable things about Google and what I perceived as a bogus witch hunt," Travis said openly. "I didn't consider it remotely likely at first that the malware report had any validity. I guess it was the result of getting so many false positives from antivirus software - especially the crummy spyware that pretends to be antivirus software and gives you a list of 500+ threats present on your system then demands money to clean it - as well as dealing with melodramatic self-signed certificate errors. When you pretty much have to arm wrestle software into doing what it's supposed to do despite a bunch of Chicken Little prompts, you get a bit jaded about so-called threats. And that's exactly the danger here."
Travis called a quick meeting with department heads to fill them in on what was going on. They reviewed the advisory from Google in the screenshot above, took note of the references to Google WebMaster Tools and Google's WebMaster Help Center, and then started delving into the issue.
The "Uh-Oh" momentThe Google WebMaster Tools page provides a link titled "Malware and Spam." (Figure E)
Travis and group investigated the section titled "Cross-site malware warnings" which explained what was happening: "When users visit a web page, browsers like Chrome check the content that's loaded to see if any part is potentially dangerous. When that happens, the browser will show a warning, alerting users that content from a site we've identified as being malicious is being loaded. In many cases, we'll also flag the original site as malicious, which alerts the webmaster and helps to protect potential users."
They accessed the "About malware and hacked sites" link, which advised taking the site offline, cleaning it to remove the malware, then requesting a malware review so Google could analyze the site and pronounce it fit for duty:
- If your site is infected: How to clean up a hacked site.
- Preventing malware infection: Best practices for avoiding infection in the future
- Requesting a malware review: Once you're sure that all spam and malicious code has been removed, you can ask Google to review it. Google will check your site and, if it's now clean, will remove any warning label that appears in your site's listing on the search results page.
"We immediately debated whether it was better to leave the site up and potentially confuse customers, or take it offline and render it completely inaccessible (and also unavailable for review by Google). At the time I was still working with the angry assumption that Google had wrongfully blacklisted us, and my first response was 'We need to get them to take us off that list right away,' without considering the part about removing the malware," Travis told me. "Then a light bulb went on over my head and I said 'Uh-oh, what if the site really was hacked?'"
To complicate the issue, one of the directors came in and reported that some internal company URLs weren't accessible on a set of handheld terminals which use Firefox to complete certain web-based functions through internal websites. There was no input hardware to click on the option to proceed, so those terminals were dead in the water, which meant the business was suffering. As can be the case with directors, constant status updates were requested, so Travis wisely posted a liaison near his cube - with whom he kept up a verbal dialogue - to inform approaching employees what was going on and let Travis work uninterrupted. As Travis put it, "there's nothing less productive or more distracting than having to stop working on an emergency response to tell people over and over what's going on. It's like a fireman putting down the hose and reporting 'I've almost got that second story blaze put out now, chief - next I plan to get on the ladder and inspect the third floor.'"
Glowstack had an existing Google WebMaster Tools account, and Travis logged in and requested a review of the site so it could be checked again for malware. The site informed him a review would commence but this might take up to 24 hours. Unfortunately, though WebMaster Tools has a support page including a forum, it was apparent there was no way to get an actual human being on the phone for immediate assistance.
The time came to make the call and pull the plug on the web server. It was taken offline but left running so that it could be examined to find out if it truly had malware. The notion of restoring the site from backup was discussed. Tragically, the DAS unit (direct attached storage) which housed the past month of the company's disk-based backups had failed two days previously, taking the last full backup of the site with it. Full backups should have run immediately once the DAS was replaced, but a hardware error prevented these jobs from kicking off. The possible malware-compromised site they had just taken offline was their one functional copy.
"That was my bad, completely" Travis said. "The backup storage unit failed, the backup job didn't kick off and the alerts failed due to a misconfigured SMTP entry. I was too busy putting out user fires to detect the carbon monoxide that posed a much deadlier threat. You can't say this was a perfect storm, because a storm can't be prevented."
Travis arranged for customer service representatives to proactively reach out to frequent business customers where possible to inform them the site was temporarily unavailable (some of these customers interact with the site on a daily basis). He then engaged a professional IT security engineer to come to the facility and inspect the web server. Ultimately, malware was indeed detected on the system, which had been compromised through a lesser-known vulnerability. The vulnerability and the malware were both fixed, and then server was pronounced clean and restored to working order. It might sound like a simple process, but it took the rest of the day and much of the night.
Once the site was back users who accessed it still received the same warnings (this was deemed acceptable by company management, which decided any site was better than no site). Travis requested another review, and within 24 hours the site was pronounced clean. It was business as usual again - except for the post-mortem and steps taken to ensure this would be a one-off episode and not part of a repeat series.
So, what happened?
Google Safe Browsing was responsible for displaying the malware warnings to users. Wikipedia states this service "provides lists of URLs for web resources that contain malware or phishing content. Google Chrome and Mozilla Firefox web browsers use the lists from the Google Safe Browsing service for checking pages against potential threats." Internet Explorer wasn't impacted by this problem since it uses a separate mechanism (SmartScreen filter) to notify users of malware-affected sites. The site malware was picked up by Google Safe Browsing, and then Chrome/Firefox users were presented with warnings.
This incident is similar to what recently happened to Netseer. On February 4th, cm.netseer.com, an advertising site, was labeled as having malware by Google.
Companies don't need to possess Google WebMaster Tools accounts for Google Safe Browsing scans to be conducted against their sites (however, WebMaster Tools can help you address problems found); all sites are rated regardless of whether they are owned by Google customers or not.
What do I do if Google reports malware on my site?
The first and foremost thing you should do is address the malware issue; Google Safe Browsing will not pronounce your site clean until it truly is malware-free. Google provides a diagnostic page you can use to get more information about malware detected on your site:
(replace mysite.com with your actual site URL)You will see a page something like Figure F.
This is similar to the screen Travis viewed for his company's website when it was stricken with malware. In the example above, useful clues are shown regarding the location of the malware and what the malware is doing.
The Google WebMaster Tools "Cleaning Your Site" page recommends the following process:
Scan your computer using an up-to-date scanning program to identify any malicious code the hackers might have added. Be sure to scan all your content, not just text-based files, as malicious content can often be embedded in images.If your site has been infected with malware, check the Malware page in Webmaster Tools. (On the site dashboard, click Health and then click Malware.) This page lists sample URLs from your site that have been identified as containing malicious code. Sometimes hackers will add new URLs to your site for their nefarious purposes (for example, phishing).
- Use the URL Removal tool in Webmaster Tools to request removal of hacked pages or URLs. This will prevent the hacked pages from being served to users.
- Report phishing pages to the Google Safe Browsing team.
- Use the Fetch as Google tool in Webmaster Tools to detect malware that might be hidden from the users' browsers, but served to Google's search engine crawler.
- Review the antiphishing.org recommendations on dealing with hacked sites (PDF).
- If you have other sites, check to see if these have also been hacked.
If you have access to your server, follow these additional steps:
- Check to see if any open redirects in your site have been exploited.
- Check your .htaccess file (Apache) or other access control mechanisms, depending on your website platform, for any malicious changes.
- Check your server logs to see when files were hacked (bearing in mind that hackers can alter your logs). Look for suspicious activity such as failed sign-in attempts, command history (especially as root), or unknown user accounts.
Don't have a Google WebMaster Tools account? Use this link to get started and add your site using these steps (the cost is free and there are many other useful options besides those geared towards dealing with malware). You'll have to verify ownership of the site ownership using a customized html file you need to upload to your site. It's a good idea to set this account up before your site runs afoul of malware; Google states via a blog post from last June that "signing up with Webmaster Tools helps us communicate directly with webmasters when we find something on their site, and our ongoing partnership with StopBadware.org helps webmasters who can't sign up or need additional help."
What have we learned today?
The morals of the story here are multifold:
- Run anti-malware software and ensure it can send alerts to the appropriate personnel.
- Make sure you review security issues for software and services you administer. Sign up for newsletters and alerts so you'll stay ahead of the curve.
- Always patch all vulnerabilities, or mitigate risks using relevant security advisories if patches aren't presently available.
- Conduct periodic security scans of your sites where possible, to detect any issues before they adversely impact the organization. Executives who object on financial grounds should be reminded that cleanup will quite likely be costlier than prevention.
- Take any and all reported malware threats seriously.
- Any problem with your backup environment (or redundancy capabilities) moves your business that much closer to the edge of a cliff. That's the kitchen fire that needs immediate extinguishing, rather than the small ashtray fires making up so much of day-to-day IT life.
- Think about whether you need to put critical sites on the same domain. In the case of Glowstack.com, their internal URLs were affected by this problem - not a big deal for desktop or laptop users, but those handheld terminals were down, something which would have been avoided if they'd pointed to a non-Glowstack.com URL such as deviceglowstack.com (for internal sites setting up an alternate domain would require minimal effort involving DNS and certificates). As a result, employees had to scramble to make up for the loss of productivity.
- Realize there is no 100% guaranteed checklist to cover all bases; you have to operate not from a reactive standpoint but from a perspective that can visualize all the components and working parts to assess potential pain points or "the next rivet to pop." This is more of a mindset than a concrete step.
Where can I get more information?
Here is a FAQ for Malware and hacked sites. Google has a blog page with helpful updates, as well as a forum discussing malware and hacked sites. Google WebMaster Tools presents a page on preventing malware infections, additionally.
Finally, Google provides a Safe Browsing API which developers can use to "permit applications to check URLs against Google's constantly updated lists of suspected phishing and malware pages."
Scott Matteson is a senior systems administrator and freelance technical writer who also performs consulting work for small organizations. He resides in the Greater Boston area with his wife and three children.