Software optimize

Android proof of concept shows remote surveillance is possible

Want to know what's inside someone's house? There's an app for that -- PlaceRaider.

After a short hiatus, James Bond's Q is back and ready to help 007 save MI6. No doubt, Q has read "PlaceRaider: Virtual Theft in Physical Spaces with Smartphones," a paper published by an Indiana University research team -- and concluded 007's trusty Minox is so yesterday.

It's sneakier and definitely less risk-averse to covertly install PlaceRaider on the spy's smartphone and let it take care of the reconnaissance for James.

Sensory malware

Far-fetched? Maybe not, the technology enabling sensory malware is available and something the research team of Robert Templeman (also an employee of Naval Surface Warfare Center), Zahid Rahman, along with team leaders David Crandall and Apu Kapadia is concerned about. I asked the team to define sensory malware:

A recent Pew research study shows nearly half of adult Americans now own a smartphone. These smartphones are capable computers with powerful sensors (cameras, microphones, accelerometers, GPS receivers, gyroscopes, and magnetometers) and are connected to billions of other people and devices through the Internet.

This means what's recorded by these sensors can be broadcast to the world -- if you grant permission. Malicious software that exploits this is called sensory malware and is an active area of research. We conceptualized the PlaceRaider attack and wondered if it might be possible to reconstruct the victim's physical environment for later reconnaissance uses by a burglar.

PlaceRaider

Click image to enlarge.

The slide above depicts the basic components of PlaceRaider and how the workload is shared between the victim's phone and the remote command and control site. The paper's description:

A mobile device is infected with the PlaceRaider App, which we assume is embedded within a trojan horse application. We implemented PlaceRaider for the Android platform, creating "remote services" that collect sensor data including images along with acceleration and orientation readings. These remote services can run in the background, independent of applications and with no user interface.

The raw data is reduced and formatted before being transmitted to the PlaceRaider command and control platform. The 3D models are generated through this platform, where the burglar can explore and exploit the model and associated images.

As you can see, the process is somewhat involved. Who am I kidding? It's very involved. Even so, I'd like to show you what happens as images move through the process.

Formatted image data

PlaceRaider collects images randomly, and because of that, the team needed to figure out how to remove low quality and/or redundant images. The team worked their magic (the paper explains the process nicely) using sensor data from the phone and image checking. The following slide is from one of the team's test runs.

I was curious; what about Android's built-in shutter noise? No surprise, the team had it figured out:

The Android Operating System requires the shutter sound be played when pictures are taken. However, Android does not require the volume be turned up so you can hear the shutter. To conceal this sound, we simply mute the phone immediately before a picture is taken and restore the volume level after the photo is taken (a split-second period of time).

This does require use of the MODIFY_AUDIO_SETTINGS permission which is typically seen as innocuous. This puts our software in control of what can be heard on the phone.

The next slide is a 2D representation of the 3D model constructed from the above set of images (please note, I didn't include all the individual images).

I valiantly read and reread the paper trying to figure out how they did that. But, it wasn't happening. So, I asked the team to help a poor journalist out:

The idea behind 3D reconstruction is multiple pictures of the same scene taken from different viewpoints give information about the 3D structure of a scene, just like the way humans can perceive depth because they have two eyes that see the world from slightly different vantage points.

The 3D reconstruction algorithms (also called structure from motion algorithms) find similar visual features across images, corresponding to multiple views of the same object, and use these matches to estimate the 3D scene structure.

Other work has shown how to use these algorithms to reconstruct tourist landmarks from photos uploaded to Flickr, for example, or how a robot can use its cameras to map out a 3D room. Our work does something similar, but uses images that were taken opportunistically instead of purposefully, and so are poorly composed and often blurry or noisy.

It appears that even with blurred images, PlaceRaider is good enough to partake in what the team calls "virtual burglary." As you can see, the image below is clear enough to allow the team to zoom in and determine the account numbers on a check (bottom right).

The implications

It's not hard to see the implications. This technology allows nefarious types to take malware to a completely new level. I asked the team about software availability and if the software was expensive:

We used widely-available technologies in the areas of image processing and computer vision for removing low-quality images and reconstructing the 3D space. And we leveraged open-source software for the generation and viewing of the 3D models.

That said; we still had to develop a certain amount of the software which implies if you have access to a computer, then you can write a PlaceRaider application without purchasing any software or specialized hardware.

From that, one can conclude access to the required software will not be an issue. I then asked the research team if they felt the bad guys would have any problems duplicating PlaceRaider or putting it to use:

Given the right expertise, attackers can easily duplicate our proof-of-concept as many of the tools are freely available. As we mention in the paper, there are some problems a practical attacker would experience that were not replicated in our experiments.

For example, in our human subjects tests we showed that good 3D models of a small office could be constructed from about an hour of phone use in the office. In real life, of course, people use their phones in a variety of different places as they go throughout their day. A real attacker would want to identify people with interesting environments worth snooping on, and only collect data from these interesting places to avoid being swamped in too much data.

Also, the 3D reconstruction algorithm we use assumes that scenes are relatively static, whereas real environments change as people and objects move over time.

The research team wanted me to give them the opportunity to rebut references that they were purposely creating malware:

First, this was an Indiana University project and the Navy affiliation is only because Robert is an employee of the Naval Surface Warfare Center in Crane, Indiana. Some of the headlines have been interesting, but not accurate in terms of characterizing the intent of the work and the Navy's role. We would like to point out our team does not produce and distribute malware. Our work identifies vulnerabilities with the hope they can be mitigated before miscreants can exploit them.

Final thoughts

PlaceRaider is a proof of concept app and as of yet, not in the wild. We all know that's not saying much; it's only a matter of time. The research team also mentioned that they worked with Android for practical reasons, but feel it would not require much effort to port the malware to iOS and Windows Phone.

Credits: The picture of 007's Minox camera is courtesy of SpyVibe. The other slides are courtesy of the Indiana University research team.

About

Information is my field...Writing is my passion...Coupling the two is my mission.

43 comments
ManlyElectronics
ManlyElectronics

What are practical applications of such technology, apart of spying ?

sarai1313
sarai1313

Hello to all the ships at sea. Flash the government has figured out how to use WIFI as radar to see in see through buildings. So give it up they are going to get it anyway. For me I don’t care it is after all my government and my country.

michaellashinsky
michaellashinsky

...the crown jewels my cats leave in the litter box every day! Sure, it is amazing stuff, and would make a great CSI-style effect in a movie or TV show, but who is going to do this to break into my house? For 98% of us, we should be more afraid of a crack-head breaking in than being the subject of this kind of spying. I got nothing worth this much cost or effort. Crimes of opportunity or crack-heads are a more realistic threat, and even then, mostly for the damage they would do looking for valuables that don't exist, (or tearing out the plumbing in the walls.) (On a separate note, it takes a very special kind of person to do $30,000 worth of damage to a building to steal $50 worth of copper...) (Edit: just noticed the Batman post above, That's why he's the friggin' Batman!)

InstructorJWN
InstructorJWN

Bruce Wayne did this int he Second batman movie. Life imitating art??

AngeloPC
AngeloPC

...can be flushed with a piece of electrical tape (or a built-in, mechanical lens cover). How productive.

Charles Bundy
Charles Bundy

Paper pointer! I guess I'm more optimistic that such pervasive sensor fusion might help mankind. Imagine if phones had temperature and pressure sensors and those could be fed into weather modeling systems. Or radiation sensors feeding into global mapping to determine environmental duress.

Michael Kassner
Michael Kassner

But, you have to admit, it would be a pain and I know I'd lose it in a day.

edjcox
edjcox

Place and opaque lens cover over your camera unitl needed....

Rderrow
Rderrow

Wouldn't a cheap fix for this type of attack from the camera standpoint anyway be adding a lens cover on the camera at least for the back camera the front would be a little different.

Wockenfuss
Wockenfuss

so that really is pretty easy to deal with. And I bet if proof of concept has been done, the black hats will have app waiting or being prepared to load on to your phone. Better cover the lens over.and be careful what you say because audio can give away much.

the_integrator
the_integrator

Finally the world is waking up. I first demo;ed this type of attack on an IBM laptop in 1999, listening in on conversations in a global bank. Reception was crystal clear even beyond the dividers in the office. Usual precautions apply.

pgit
pgit

More scary news, another arrow to the heart of privacy. As for "A real attacker would want to identify people with interesting environments worth snooping on, and only collect data from these interesting places to avoid being swamped in too much data." I suppose GPS would help, eg only enable sensing when within a specified radius. Fascinating stuff.

Charles Bundy
Charles Bundy

"Stupid vision tricks". Or you need a hook to perk up interest in your paper for citation :). The real meat was in proving that you didn't need highly controlled rigs and calibration to snap images for the Surface from Motion software/algorithms they already had. That and that you could generate 3D point clouds from really lo-res image sets. Practical applications would be in automation E.G. robot navigation and kinematics. The paper mentioned it was possible to do SFM on the phone, but it would have snapped up so many resources as to render the phone incapable of doing any other task. ADDENDUM: The other practical application is using exisiting information which is overwhelming "lots 'O images" into an intuitive point-n-click interface. E.G. Big Data graphical frontend.

Michael Kassner
Michael Kassner

The only references I have of practical does not refer to applications. Can you please help me, by point out where you taking the quote from.

Michael Kassner
Michael Kassner

I've been writing here for a long time now and enjoy it when members start a conversation that differs from the topic of the article. My apologies, but I sense this divergence is not one that will end well. My favor is to respectfully ask each of you to agree to disagree and leave it at that.

pgit
pgit

"it is after all my government and my country" The same attitude has allowed every tyranny the world has ever known. You are the 80% who "have eyes but do not see, have ears but do not hear" Jesus spoke of. "Government" is NOT "my country," the two do not equate, they are polar opposites. People are led by the nose from birth, through "public education" to equate "government" with "society." Governments traditionally seek to enslave, command and control societies to their own (government's) gains, and always most detrimental to the people, in the long run. Learn some history.

Michael Kassner
Michael Kassner

But, what if you left a check out or some sensitive private information. What is the cost of that? And, the real intent of the article was to increase awareness. I am glad you read the piece and now know about it. It allows you to make an informed decision.

Charles Bundy
Charles Bundy

in terms of time and money? What is your definition of [i]productive[/i] in an applied research situation? BTW if you read the paper you would see that a lot of what they used was OTS except for the Android client end parser.

Michael Kassner
Michael Kassner

What if the research team didn't put forth the effort -- only the bad guys would know.

Michael Kassner
Michael Kassner

That is a good idea. I always prefer a positive take on things. But feel it's important to make everyone aware of what's out there.

ManlyElectronics
ManlyElectronics

And put chewing gum on the microphone every time when finish the call. And if you in US then every time unswerving you have say first - "this call may be recorded by anyone".

Michael Kassner
Michael Kassner

Or a physical switch, which I have heard mentioned by sources.

Michael Kassner
Michael Kassner

I wonder if the phone OEM or provider changed the Android setting.

Michael Kassner
Michael Kassner

The ability to develop a 3D reconstruction is new, particularly when it does not require a monster super mainframe to create.

Michael Kassner
Michael Kassner

That something can become that usable from random images is amazing.

ernied
ernied

I think you're considering the 3-D modelling of the collected data as being too resource intensive for the phone, not the collection of the data. Collecting the actual data is a piece of cake. Take video of surrounding area, and add rough positional data. That is, if you even really need that positional data in the first place. If you just want to spy on your target and get as much information as possible about things in the room and potentially theft-worthy information on their desks, you just need to take video without even bothering with positional data. But then, once you've broken into your target's phone, you can also do things like find out the wireless password for the corporate network, and then snoop the network from afar (or even from the phone), collecting passwords, financial information, or any amount of other data that's being sent on the wireless connection.

sarai1313
sarai1313

What ever. Oh by the way I was not the one who got off subject. I just will not let some one,any one slam my country with paranoid rants. Because the next thing you hear is that they are already doing things that are not happening. I come hear for tech not politics. If I did I would get back in game but would not bring it up here. Peace to all from a very old independent voter. So who ever you want just make sure you do go and vote. Arguing gets you no were. Vote .

sarai1313
sarai1313

Led through my nose. I don’t need to hear from some unintelligent ass . Trying to tell me a values are wrong. Do you have a home ,a job and freedoms that only you can get here in America. Then shut up. I have never broke the law,never done anything against my country and my government I have voted in to office. Drone I have served my country at time of war and peace. Served my community,and my church. . All I hear is some one crying that the government is bad it is out to take are freedoms away by spying on us. You know what kid you sound like all the others. Oh the government is out to get you crap. Don’t like it change it. But don’t sit in your mothers basement and think you know how the world thinks. I wont even tell you who I am ,were I am from,who I work for,or my credentials. I am one of the folks who do rebel against the machine. I have made changes In the way my government works and to be honest the only thing I like to change is the draft. So you folks who say my government is out to get you would be right . But then again you yes you would run to Canada. You want to reply go for it.

AngeloPC
AngeloPC

What if the "research team" ARE the bad guys?

Charles Bundy
Charles Bundy

Per 1990s specifications when I was working in active vision. Just so happens we all have them on our desks or entertainment centers now! :)

pgit
pgit

Definitely the most fascinating aspect. I wonder if there is any input from motion sensors/gyros or GPS in the algorithms that generate the 3D view... That would make it all the more amazing by a factor of ten.

Michael Kassner
Michael Kassner

That's the weak link in any kind of security, it always reactive.

Michael Kassner
Michael Kassner

The card in theCAD units has a huge fan as well and is noisy.

Charles Bundy
Charles Bundy

Pixel pipeline unit hooked up to a dedicated Sun box. As I recall it had a whopping 512 linear processing units. I just noticed that you can get Nvidia cards with three times the pixel units and about 1000 times the memory for $299 on newegg. :)

Michael Kassner
Michael Kassner

I just setup several CAD stations that are for solid-modeling videos. The video card had four times the memory of my notebook.

Michael Kassner
Michael Kassner

I can see the relationship. And, what the software accomplishes is impressive.

Charles Bundy
Charles Bundy

Inertial Measurement Unit and Multiple Image Structure from Motion. What these folks are doing is using your mobile device as an Xbox type kinect sensor, just not in real time. You get the same 3d point cloud data that can be explored with an appropriate viewer. When you see something interesting you can click on the point cloud and the relevant set of images are displayed (ala check number example.) That's what they mean by virtual theft and exploration. Its really a virtual environment.

Charles Bundy
Charles Bundy

On the device side in paring down duplicitave / non useful images for the backend point cloud generation. Otherwise a user might notice they were infected by their bill for data transmission. :) I didn't see any IMU type data in the paper and the tools they used for MISFM wouldn't be able to use it.

Michael Kassner
Michael Kassner

The paper goes into the details, but hang on. It's quite a ride, at least it was for me.