After a short hiatus, James Bond’s Q is back and ready to help 007 save MI6. No doubt, Q has read “PlaceRaider: Virtual Theft in Physical Spaces with Smartphones,” a paper published by an Indiana University research team — and concluded 007’s trusty Minox is so yesterday.

It’s sneakier and definitely less risk-averse to covertly install PlaceRaider on the spy’s smartphone and let it take care of the reconnaissance for James.

Sensory malware

Far-fetched? Maybe not, the technology enabling sensory malware is available and something the research team of Robert Templeman (also an employee of Naval Surface Warfare Center), Zahid Rahman, along with team leaders David Crandall and Apu Kapadia is concerned about. I asked the team to define sensory malware:

A recent Pew research study shows nearly half of adult Americans now own a smartphone. These smartphones are capable computers with powerful sensors (cameras, microphones, accelerometers, GPS receivers, gyroscopes, and magnetometers) and are connected to billions of other people and devices through the Internet.

This means what’s recorded by these sensors can be broadcast to the world — if you grant permission. Malicious software that exploits this is called sensory malware and is an active area of research. We conceptualized the PlaceRaider attack and wondered if it might be possible to reconstruct the victim’s physical environment for later reconnaissance uses by a burglar.


Click image to enlarge.

The slide above depicts the basic components of PlaceRaider and how the workload is shared between the victim’s phone and the remote command and control site. The paper’s description:

A mobile device is infected with the PlaceRaider App, which we assume is embedded within a trojan horse application. We implemented PlaceRaider for the Android platform, creating “remote services” that collect sensor data including images along with acceleration and orientation readings. These remote services can run in the background, independent of applications and with no user interface.

The raw data is reduced and formatted before being transmitted to the PlaceRaider command and control platform. The 3D models are generated through this platform, where the burglar can explore and exploit the model and associated images.

As you can see, the process is somewhat involved. Who am I kidding? It’s very involved. Even so, I’d like to show you what happens as images move through the process.

Formatted image data

PlaceRaider collects images randomly, and because of that, the team needed to figure out how to remove low quality and/or redundant images. The team worked their magic (the paper explains the process nicely) using sensor data from the phone and image checking. The following slide is from one of the team’s test runs.

I was curious; what about Android’s built-in shutter noise? No surprise, the team had it figured out:

The Android Operating System requires the shutter sound be played when pictures are taken. However, Android does not require the volume be turned up so you can hear the shutter. To conceal this sound, we simply mute the phone immediately before a picture is taken and restore the volume level after the photo is taken (a split-second period of time).

This does require use of the MODIFY_AUDIO_SETTINGS permission which is typically seen as innocuous. This puts our software in control of what can be heard on the phone.

The next slide is a 2D representation of the 3D model constructed from the above set of images (please note, I didn’t include all the individual images).

I valiantly read and reread the paper trying to figure out how they did that. But, it wasn’t happening. So, I asked the team to help a poor journalist out:

The idea behind 3D reconstruction is multiple pictures of the same scene taken from different viewpoints give information about the 3D structure of a scene, just like the way humans can perceive depth because they have two eyes that see the world from slightly different vantage points.

The 3D reconstruction algorithms (also called structure from motion algorithms) find similar visual features across images, corresponding to multiple views of the same object, and use these matches to estimate the 3D scene structure.

Other work has shown how to use these algorithms to reconstruct tourist landmarks from photos uploaded to Flickr, for example, or how a robot can use its cameras to map out a 3D room. Our work does something similar, but uses images that were taken opportunistically instead of purposefully, and so are poorly composed and often blurry or noisy.

It appears that even with blurred images, PlaceRaider is good enough to partake in what the team calls “virtual burglary.” As you can see, the image below is clear enough to allow the team to zoom in and determine the account numbers on a check (bottom right).

The implications

It’s not hard to see the implications. This technology allows nefarious types to take malware to a completely new level. I asked the team about software availability and if the software was expensive:

We used widely-available technologies in the areas of image processing and computer vision for removing low-quality images and reconstructing the 3D space. And we leveraged open-source software for the generation and viewing of the 3D models.

That said; we still had to develop a certain amount of the software which implies if you have access to a computer, then you can write a PlaceRaider application without purchasing any software or specialized hardware.

From that, one can conclude access to the required software will not be an issue. I then asked the research team if they felt the bad guys would have any problems duplicating PlaceRaider or putting it to use:

Given the right expertise, attackers can easily duplicate our proof-of-concept as many of the tools are freely available. As we mention in the paper, there are some problems a practical attacker would experience that were not replicated in our experiments.

For example, in our human subjects tests we showed that good 3D models of a small office could be constructed from about an hour of phone use in the office. In real life, of course, people use their phones in a variety of different places as they go throughout their day. A real attacker would want to identify people with interesting environments worth snooping on, and only collect data from these interesting places to avoid being swamped in too much data.

Also, the 3D reconstruction algorithm we use assumes that scenes are relatively static, whereas real environments change as people and objects move over time.

The research team wanted me to give them the opportunity to rebut references that they were purposely creating malware:

First, this was an Indiana University project and the Navy affiliation is only because Robert is an employee of the Naval Surface Warfare Center in Crane, Indiana. Some of the headlines have been interesting, but not accurate in terms of characterizing the intent of the work and the Navy’s role. We would like to point out our team does not produce and distribute malware. Our work identifies vulnerabilities with the hope they can be mitigated before miscreants can exploit them.

Final thoughts

PlaceRaider is a proof of concept app and as of yet, not in the wild. We all know that’s not saying much; it’s only a matter of time. The research team also mentioned that they worked with Android for practical reasons, but feel it would not require much effort to port the malware to iOS and Windows Phone.

Credits: The picture of 007’s Minox camera is courtesy of SpyVibe. The other slides are courtesy of the Indiana University research team.