Smartphones

Google Play: Android's Bouncer can be pwned

Deciding whether to trust apps or not just got more complicated. Michael Kassner asked a pair of researchers why that is.

I can't believe my good fortune. I'm interviewing two Pwn2Own legends about their latest tag-team exploit; owning Android's anti-malware system -- Bouncer.

What is Bouncer?

When Google first introduced their app store -- Android Market -- in 2008, apps weren't tested for malicious content. Fellow TechRepublic writer William Francis and I verified that in this article about Android Permissions.

Google rectified their oversight this past February with the introduction of Bouncer:

"Today we're revealing a service codenamed Bouncer, which provides automated scanning of Android Market for potentially malicious software without disrupting the user experience of Android Market or requiring developers to go through an application approval process."

So that's it, no more malicious apps in their app store. Or is it....

Who are these legends?

Dr. Charlie Miller and Dr. Jon Oberheide, that's who.

I'll start with Charlie. Normally, it's enough to say someone worked for the National Security Agency, but he was just getting started. I mentioned Pwn2Own earlier; Charlie has won at Pwn2Own four years in a row.

My favorite "Miller" exploit surfaced last year. Charlie planted a "sleeper app" in Apple's App Store. The app exploited a vulnerability that allowed Charlie to send "other than Apple-approved" commands to devices that downloaded his app. Charlie alerted Apple about the vulnerability ahead of his going public with it at Pwn2Own, but neglected to mention the sleeper app. Apple didn't take kindly to his omission, revoking his developer's license.

Next is Jon. He's the serious one on the left. Currently, Jon is co-founder and CTO of Duo Security. The company provides two-factor authentication as a service. When Jon is not working at Duo Security or with Charlie, he's sneaking around trying to find exploits.

I remember last year's Pwn2Own -- Jon probably would have won. But rather than save his Android exploit for the contest, Jon reported it to Google, netting himself $1337 US. That's nice, until you consider Jon could have won $15,000 US if he had waited. I'll let Jon explain.

So what's with Bouncer?

Charlie and Jon are able to bypass Bouncer and slip malicious apps into Google Play. (Note: I don't know about you, but I'm confused as to whether it's Google Play or Google Play Store. For now, let's agree to use Play, even though on my phone, it's Play Store.)

Charlie and Jon presented Dissecting Android's Bouncer (Warning: Charlie is exposing more than code in the pdf.) at SummerCon 2012. After reading the transcript, I contacted my heroes to see if they would agree to an interview. And they were.

Kassner: Hello, Jon. Thanks for helping me understand what you and Charlie accomplished. Why did you decide to look at Bouncer in the first place? Oberheide: Literally, a couple days after the public announcement of Bouncer, I decided remote blackbox analysis of Bouncer would make interesting research. In a previous life, I worked on malware analysis, both writing and breaking sandboxes, so this was an area relevant to both my past and current interests.

Charlie and I are always bouncing mobile stuff back and forth. So we decided this would be a fun project to work on together and present at SummerCon.

Kassner: If I understand correctly, you are the first ones not associated with Google to figure out how Bouncer works. Can you give us an overview of what you found? Oberheide: We started with a completely blackbox perspective, knowing nothing about how the Bouncer system operated. We then created a list of questions we hoped to answer about how Bouncer works and what potential weaknesses are present.

If I were to sum up our discoveries, I'd claim that Bouncer is currently in a 1.0 state. Which is fair, given it was only recently publicly announced. We discovered several ways to fingerprint the Bouncer environment. The most interesting ones would allow a malicious app to hide its true intentions while running with Bouncer, and still perform malicious activity on a real user's device.

So, there's a lot Google needs to do to address Bouncer's weaknesses and advance its capabilities. That being said, I wouldn't be surprised if Bouncer is fairly effective at cleaning up the current generation of mobile malware, which tends to be unsophisticated.

However, as mobile devices become more enticing targets and attackers find it economically worthwhile to put effort into targeting mobile users; we'll see an increase of sophistication to sneak past Bouncer. If there's anything we've learned from traditional malware, it's that attackers will adapt quickly, and they're quite efficient at sharing (or stealing) techniques.

Kassner: I'm curious as to why Bouncer allows the app under test to access the Internet. Couldn't Bouncer fake it? That would prevent you from communicating with Bouncer. Right? Oberheide: Absolutely, there are a number of networking approaches one can take when designing a dynamic-analysis system: blocking, filtering, emulating (aka "faking" like you mention), and unrestricted network access are examples. Problem being, each approach has its pros and cons.

If the designers of Bouncer had decided to fake network access, it would have been difficult to do in such a way that is indistinguishable from real network access. But you're right; the available network access was useful for leaking information to our command and control servers.

Kassner: I read you were playing cat-and-mouse with a real person and not just the automated Bouncer process. How did you determine that? Did that person shut you down? Oberheide: During some of our more aggressive experiments, we noticed periodic "callbacks" -- the app phoning home to us from within Bouncer -- coming from a different Google IP range. And the callbacks exhibited request patterns and probes that indicated a human operator was remotely analyzing the app.

We hypothesized this was part of the overall Bouncer workflow. That is, if an app is flagged by the automated dynamic analysis of Bouncer, it would trigger a manual review by a person.

One time when I submitted an app that opened a remote shell into Bouncer, I didn't catch the warning signs I mentioned earlier and proceeded to send commands to the shell. When that didn't work, I had an idea why. So I tried to send the person on the other end a few friendly messages.

Kassner: Going through the presentation, I noticed this slide.

"We can submit apps without paying!"

I also read you've informed Google about your findings. So how you were able to submit apps without paying?

Oberheide: One of our main concerns during our Bouncer experimentation was to keep real users out of harm's way when we submitted our apps (our Hippocratic Oath: Do no harm).

Thankfully, we discovered Bouncer would analyze the submitted app after it was saved in the Play publisher interface, but before it was made available to the public. So we could perform our experiments with Bouncer without affecting users.

The icing on the cake was our ability to upload applications and have them analyzed by Bouncer, even if Google Play denied our payment. Obviously, you can't publish apps if you don't pay, but we didn't need to go that far, this was a boon for our research.

Google should address this issue. Attackers could probe Bouncer without having to use a legitimate credit card number.

Kassner: I've read Apple validates apps through static analysis (checking object code) and through your efforts, we know Google is using dynamic analysis (running the app). Which is better? Oberheide: I've never seen any evidence published that Apple performs any security review of applications, dynamic or static, so I can't comment authoritatively on that. It appears that Apple may use static analysis to check for the use of private APIs, but that's hardly indicative of whether an app is malicious or not.

Generally, there are pros and cons of both static and dynamic analysis, depending on the threat model and characteristics of the target platform. For example, static analysis would be a bit easier for Apple, since they perform code signing. Code signing gives them higher confidence that the code submitted along with the app is the only code that will be executed in the app's lifetime. It's very difficult for Google to perform static analysis since Android allows additional code to be pulled down at runtime changing the behavior of an application.

More importantly, static and dynamic analyses are not mutually exclusive. Any good system tasked with analyzing potentially malicious software will employ a combination of both techniques.

Kassner: In the presentation, you mention there's an individual's Google account associated with Bouncer:

Google account associated with the Bouncer device:

  • base64.b64decode('OyBtaWxlcy5rYXJsc29uQGdtYWlsLmNvbSwgY29tLmdvb2
  • dsZQ==')';XXXXXXX@gmail.com,com.google'
  • XXXXXXX@gmail.com

And, you found interesting tidbits about this person, such as the contact list of the account holder. Why store personal information in Bouncer?

Oberheide: In order to make Bouncer effective, it must be indistinguishable from a real user's mobile device. Otherwise, a malicious application will be able to determine it is running with Bouncer and not execute its malicious payload.

As you can imagine, creating a fake device that appears to be real is a difficult problem. A real mobile device has a variety of data sources (phone book, SMS messages, photos, music, third-party applications, etc.) and sensors (microphone, camera, GPS, accelerometer, etc), many of which are difficult to fake in a believable and scalable manner.

Kassner: As I read the presentation, I noticed that you were keeping track of all the ways a malicious app developer could deceive Bouncer.

Bouncer runs your app for 5 minutes

  • Don't do anything bad for 5 minutes! Duh.
  • Not long term. Could be run later, longer...

A while back, I wrote a piece about Bouncer and how people were finding ways to circumvent it. From what you know now, should we be concerned about the apps in Google Play? What safety precautions would you suggest we take?

Oberheide: Yep, I had the same thoughts (regarding circumvention) when it announced. The RootSmart malware Dr. Jiang discovered was based on the RootStrap approach I had proposed two years ago at SummerCon 2010.

It took malware authors almost two years to adopt that technique...but then again, there wasn't much "evolutionary pressure" for them to do so until Bouncer came around.

Users should always be cautious about downloading suspicious apps from an app store:

  • Regardless of the efficacy of Bouncer.
  • Regardless of what mobile platform they're using.
  • Regardless if "antivirus" is installed on their mobile device.

As we've seen in the desktop-computing world; no malware protection, whether gated at the app store or performed on-device, will be entirely effective. That said, it's still worthwhile for Google to block as much malware upfront as possible, and I'm sure they're hard at work revising Bouncer to address the weaknesses we discovered.

Final thoughts

Charlie and Jon are best of the best when it comes to finding vulnerabilities. But I fear others may not be far behind. Let's hope Google is hard at work removing this vulnerability.

Jon wanted to make sure I attributed the awesome high-def slides in this article to Busticati Productions. (I think we've been busticated.)

About

Information is my field...Writing is my passion...Coupling the two is my mission.

22 comments
Ocie3
Ocie3

*who* was the other person that they encountered during their research? It wasn't necessarily someone on the Google payroll, it seems to me.

peter.stephens
peter.stephens

Hi Michael, thanks again for another great article. Most always enjoys your stuff. Very interesting to see the blending of automated risk flagging with a human review step. Probably the only way to be thorough though the fact that Android apps can always pull down new code means that if an app developers update mechanism was compromised, we would be in the do do.

mike
mike

I fear that all that Bouncer debate leads to some kind of wrong questioning. In my opinion, the first question should be whether to trust Android. The deep integration of various Google services in this system makes me feel very skeptical about it???sprivacy. My tablet is pretty nice, but even without installing any third party app, data leakage is so obvious, that the real question could be: Should we trust Google?! Don't misunderstand me: I don't think Google peeks on us more than some other. They all do, and they all become greedier on our personal data every day... Of course, my question should be asked for every operating system, every software, and every service-of-any-kind provider we use, especially if our data could be stored (sometimes without our knowledge or consent) in the cloud. I do not have answers, but in the meantime I never transfer any personal or sensitive information of any kind on my Android (nor on some other non-Android) devices. Be a paranoid and your data will not be "gone with the wind"...

Michael Kassner
Michael Kassner

New Post Bouncer is supposed to prevent malicious apps from being placed in Play Store. Two legendary researchers explain how Bouncer can be circumvented

Michael Kassner
Michael Kassner

I found the human touch interesting as well. My next thought was how often is it required?

JCitizen
JCitizen

I find my self trusting MS more and Apple/Google less and less each day. Redmond probably doesn't deserve it, but they haven't been caught with their hands in the cookie jar lately. Or at least it has not be published lately. This is just a personal perception of mine. No basis in reality, of course. :)

AZ_IT
AZ_IT

of our personal devices, from hardware manufacture to OS and app software, to the network it runs on there's not really a way to ensure our data remains on the device. From country spying on country to corporation spying on corporation anything could be embedded into the hardware or software. It comes down to trust. Intel has banned the use of siri by employees. Are they being paranoid? Yes but possibly with good reason. It would be a relatively small matter for Apple, Google, or even Microsoft to find someone's cell phone number, identify the actual device, and monitor communications to that phone. I don't think I'm important enough to warrant that amount of attention and for the most part as a consumer I trust that the devices are only being used as advertised. That is possibly naive. But I don't have every box of Cheerios I open analyzed to make sure it's not poisoned or accidentally contaminated either so I guess I like to live on the edge. Eventually it comes down to simple cost to benefit ratios. The benefits of using the device out-weigh the potential costs whether real or imagined. Having a smart phone is just too blasted convenient.

Michael Kassner
Michael Kassner

We can be as safe and paranoid as we can, but unless you are completely off the grid: no doctor, SSN, credit card, etc. Someone can break in a database somewhere and possibly get your info.

AZ_IT
AZ_IT

Thanks for another interesting article Michael. I always enjoy them. I've also heard that Apple reviews the code of all the apps in the app store so I thought it was interesting to hear that their static security review couldn't be confirmed by a major vulnerability researcher. Then again I don't buy into all the hype around Apple products either. If it looks like a computer and it's built like a computer (CPU, RAM, Storage, Network) then it has as many vulnerabilities as every other computer. Then it's only a question of how to harden the device appropriately through software. The only reason a calculator is any more secure than a computer is because it isn't connected to the internet so the avenues of attack are purely physical. There is nothing god-like about Apple computers. Having said that they do hand-select high-quality hardware and do a better than average job of testing it all together so that's a nice plus. But besides that the only other thing they have is the hype and it never fails to amaze me how many people buy into it (I have multiple coworkers who are Apple drones which is why it gets me so fired up). The problem with all app stores is that unless you are writing everything yourself there is really no way to ensure that the apps are all malware free. Can and should security and review processes be better? Of course. But history has proven that for every stride we make in security the other side finds a way to circumvent it or simply finds a new avenue of attack. I guess my point is there are no silver bullets in the security world. There are only ways to mitigate the threat. Bouncer seems a good stride in the right direction but still has room for improvement.

mark
mark

and the justice department stepped back. When Microsoft wasn't lobbying and they were making $$$ but not giving millions and millions to the Justice Department they were getting investigated for anti-trust on a monthly basis. Then MS started the PAY OFF to our government thugs and mysteriously they were automagically not in hot water any more that is until the European governments found out how much the Microsoft pay off to the US government was. Now that Microsoft pays off most of the big governments in the world they are once again "compliant" and we havent heard of much Anti-Trust suits against MS. When government gets too big countries and corporations suffer. EVERY TIME!

AZ_IT
AZ_IT

is that they've made a lot of the mistakes these other companies are now making: Apple with it's patent wars, google with it's intrusiveness as two prime examples. Apple and Google will probably mature with time. Of course it could be that Microsoft just got better at hiding it's illicit activity but I'm hoping they outgrew it instead. I think the fear with Google may be based mostly on the fact that they have rather spectacularly managed to index the internet. That doesn't necessarily translate into them indexing our data although it easily could.

Michael Kassner
Michael Kassner

Check out what the hot IT job is now. It's database experts. They glob all the data together and using magical algorithms put two and two together.

AZ_IT
AZ_IT

It was also great to hear that there is a human component involved in the review process as well.

Michael Kassner
Michael Kassner

Jon mentioned he was confident Google was already fixing the oops they found.

Michael Kassner
Michael Kassner

All one has to do is try and find something that Microsoft hasn't affected to realize...

AZ_IT
AZ_IT

And to be honest I'm kind of intrigued just to see how the product will eventually materialize and to see how people actually use it. I'm with most of the nay-sayers in thinking it will be a very niche market of people that ever buy it. But as anyone who wears corrective glasses can tell you it's a lot harder to walk out the door without your glasses on then it is to walk out the door without your phone. Microsoft has been investigating numerous ways to make screens and circuits bendable possibly even wearable. I think the concept has promise I'm just not sure Glass will be the next iPod. I personally love the concept of the Microsoft Surface (rebranded as PixelSense). But it hasn't materialized into a consumer device and with the cost it may never. Glass to me will probably go the same way. I'm still hopeful for both products though.

Michael Kassner
Michael Kassner

I like the bit about "Glass" Did you see the intro video about it?

AZ_IT
AZ_IT

What the heck?!? It's just unfortunate that they probably had no mechanism to reply. Even with only inbound communication I'm sure it made for entertaining office banter. Google employee Phelps: "You'll never believe what just happened. One of the apps I was reviewing started talking to me." Google supervisor: *Chuckles* "Oh really, what did it say?" Phelps: "Well it asked how my day was going. I wasn't quite sure how to respond, I tried typing a reply of 'I'm fine. How are you?' but my command prompt told me that 'I'm' is not recognized as an internal or external command. So then I tried talking to it but of course it didn't respond." Supervisor: "Interesting approach. Did it say anything else?" Phelps: "Actually it did. It complimented my shirt. Then it told me I looked like a trustworthy sort and proceeded to explain how it was an alien intelligence trying to warn us of a take-over plot. It told me that the app was totally safe and to allow it into the market so it could warn more people. I almost did but then I noticed that their payment never went through." Supervisor: "Did it share anything else?" Phelps: "No it just stopped talking after that." Supervisor: "I think we may be working you too hard. Maybe you should take tomorrow off." Later over the phone. Supervisor to Schmidt: "We have a leak in our task force. I think Phelps may suspect." Schmidt: "We must accelerate our plans then. Glass must launch this year!" Happy Tuesday.

Michael Kassner
Michael Kassner

I was trying to imagine what that person thought when the messages showed up.