Mobility

Smartphone apps: They may be listening

Is your smartphone listening? It might. And, you may not even know it.

You walk into a store. There's a special device transmitting on a frequency humans can't hear. But smartphones can. And, thanks to that new app you downloaded, it is listening. Why? The retailer wants to know who you are and that you're in the house.

Deal, or no deal?

Incentivize the public

A friend called, all excited about something called Shopkick. He gushed on about saving money when shopping at Best Buy. That's a good thing, as he spends prodigious amounts of time and cash there.

It seems, Shopkick is:

"The first mobile app that gives you rewards and offers simply for walking into stores, for scanning products, and for signing up friends. You can collect kick bucks and bonuses at millions of stores and restaurants in America."

My friend asked if I had heard of Shopkick. I mentioned that I didn't, knowing full well it was his way of getting me interested. The call ended with an ultimatum.

"Don't take long. My brother borrowed my speakers for a party and trashed the sub. I need a new one bad."

That's my friend, selflessly helping the economy at every chance.

Seemed normal

I thought, why not see where this leads. I started with Shopkick's website. I noticed that several big-name retailers were involved. And, media outlets were saying nice things about Shopkick. So far, so good.

Next, I checked their Terms of Service and Privacy Policy: nothing unusual. It was time to get serious. I didn't find much initially. Then, I came across a This Week in Tech (TWiT) podcast by Leo Laporte, where Shopkick was one of the topics being discussed.

As I listened, it got interesting real fast.

Surprise, surprise

Mr. Laporte and the panel of guests were discussing smartphone apps and why the apps were turning the microphone on. That certainly grabbed my attention. Especially, Mr. Laporte asking:

"Does it scare anybody we've learned that this program (referring to Color) turned on the microphone and was listening? Now, they're not using it in any nefarious way.

But they didn't have to tell us. They just did it. Doesn't it mean that there may be many other apps that are doing the same thing? Doesn't that bother anybody?"

I'll get to this later. First, I want you to read what panel member, Robert Scoble points out:

"By the way it's not the only app (Color) that uses the microphone. Shopkick uses the microphone to know when you entered the store. You turned on Shopkick and it's actually listening for an inaudible signal.

So they have a speaker in front of Macy's that when you take your iPhone inside Macy's, inside the front door, it senses the audio signal and gives you points for entering the store."

Not mentioned in EULA

Remember my not noticing anything unusual (like using the smartphone mike) in the Shopkick Terms of Service and Privacy Policy? Mr. Laporte reasons why: "They didn't have to tell us. They just did it."

I felt it important to understand what the developers had in mind and why it wasn't mentioned on their website.

To that end, I contacted Shopkick. Ms. Katie Carlson of Atomic PR helped make the connections. Cyriac Roeding, cofounder and CEO of Shopkick was kind enough to answer the following questions.

Kassner: The use of the phone's microphone is not mentioned on the website. Or not easily found, if it is. Why is that? Roeding: Shopkick's main focus is consumers, so the information on the site is geared towards helping them get the most out of the application.

We are very public about how the technology works. In fact, we believe this technology is a major breakthrough in location technologies. You will find many articles in which we talk about technical details.

We take user privacy very seriously. Users need to open the app, in order to activate the technology, so they are 100% in control. It's like a satellite GPS signal. You only pick it up when you want to. We don't do anything with the audio signal except try to identify a signal the app can understand. And, none of the audio itself gets stored.

Kassner: Can you explain how the Shopkick app uses the microphone? Roeding: The microphone is used as part of our location technology called the ‘Shopkick Signal,' which enables the app to verify presence within a store or mall.

Stores have transmitters on-site that emit inaudible audio, which is picked up by the microphone on a user's smartphone, when (and only when) the Shopkick application is open.

Because the detection occurs on the phone rather than by the store, privacy of presence information is completely under the users' control.

Kassner: What does the app do with the captured Shopkick signal? Roeding: As for the application, it samples the ambient audio through the microphone and isolates the Shopkick signal, which appears in inaudible portions of the audio spectrum. This is analyzed using digital signal processing technology to decode the signal. Kassner: What assurances do people have that the audio is indeed unusable? Roeding: We don't store or transmit recordings of the audio itself, only processed information about the Shopkick signal.

Final thoughts

When I told my friend what Shopkick does, he was able to make an educated decision on whether to use it or not. Without significant searching that would not have been possible.

I'd like to thank Mr. Laporte, Mr. Scoble, Mr. Roeding, and Ms. Carlson for helping me get to the bottom of this mystery.

About

Information is my field...Writing is my passion...Coupling the two is my mission.

116 comments
bboyd
bboyd

though many others write well of the future world. Real science fiction is about the technology impact on humanity not just a setting to play in. It really is amazing what an extension of what to us seems to be a untested and minor technology but ends up changing the way humans live in a broad sense. Imagine the disasters wrought by a cure for cancer, or the freedom given by instant mental communication. Inversely the hope and joy that cure brings and tyranny and domination a direct contact might wreck. Edit: Should have been reply to MK's 1984 comment..

pgit
pgit

I mentioned this in a prior post: http://www.thenewspaper.com/news/34/3458.asp Routine traffic stops. This is illegal and unlawful, (big difference there) they are "fishing" for anything to pin on you, and pin they will. We are beyond the brownshirts, folks. A lot of our "law enforcement" is in fact waffen SS. Armed heavily and gunning for YOU. (just whom do they "protect and serve" by stealing your cell phone or laptop contents?) It's gonna get ugly before it gets any better, IF it ever gets better again.

polycom
polycom

I got an app called Task Identifier on Android and apps load at weird times. Make a phone call and my eBay loaded... what could they want or sitting idle and my video recorder loads, constantly... they listen...

pgit
pgit

OK, having read down to the bottom of this thread I've concluded that if I get a mobile device I'm making a sealed lead sheath for it. I have a 300 lb slab of pure lead in the garage, used to be ballast for balancing aircraft loads. You can hammer that stuff pretty thin. I could sew some between a couple pieces of leather and make myself an electromagnetically, visibly and acoustically isolated environment for it. Of course it wouldn't be receiving incoming calls, but what's voice mail for then?

strandley12
strandley12

Some apps are helpful while some are too much. Some help track down people while some let you know where you are. and some are fun while others are dangerous and crazy.

hiraghm
hiraghm

How does this signal affect small animals? My mother goes nowhere without her chihuahua, and I'd hate to think of the carnage that would ensue if I discovered they were putting out a signal that caused her pain or discomfort.

mtndive
mtndive

I think that phone OSes need to add better access to the permissions that an app requests/has. A security setting that controls which apps can use a service on the phone. The GPS for example. I can turn the GPS/Geolocating service off. Yes, I know emergency services can turn it back on. However, it still prevents apps from using my GPS in my phone to find me (except for the good hackers) - it also means that I have to turn it on to use my mapping software. Unlike GPS, which has a good reason for emergency services to turn it on, the microphone and camera service wouldn't require that exception. If the use of the microphone and camera were permissions that had to be allowed by the user when they install a program they would be aware of the intention to use it. Such a flag should be put up by the OS and not the app.

AnsuGisalas
AnsuGisalas

Think about all the crapware apps (Crapps) out there... any one of them could have this capability to eavesdrop on command as a rider or payload. Industrial espionage and social engineering would be obvious applications. I'd like to know what the phone manufacturers have to say about this capability, and if it would be possible to lock down phone capacity for opening the mic without user intervention or permission.

Michael Kassner
Michael Kassner

Is my approach. You bring to bear a very interesting point. Science fiction was amazing in my youth, now it it scary.

Michael Kassner
Michael Kassner

And they are all over that. From what I have read there is no prior law. So, it will have to work its way through the system.

Michael Kassner
Michael Kassner

Thanks for bringing them to our attention. I have not dealt with Task Identifier yet.

apotheon
apotheon

Wire mesh (Faraday cage) works fine for everything except the microphone and camera. Put tape over the camera and seal the microphone with epoxy, and you're golden -- except you won't be able to talk to anyone on a telephone call without unsealing the microphone.

Michael Kassner
Michael Kassner

How you maneuver the lead around. And, what other strange and wonderful things do you have laying around.

Michael Kassner
Michael Kassner

To a portion of the populace. The question that always come to mind is: at what price. Edit: Replace populous with the correct form: Populace.

Michael Kassner
Michael Kassner

A hardware switch for the microphone. GPS is one thing, but capturing conversations is more intrusive, IMO.

mcbriendl
mcbriendl

Spam....really, on a tech site? There ought to be a law.for "people" like you.

pgit
pgit

I wondered why the likes of Apple were so adamant about not tolerating 'rooting' your device. I'm sure the excuse is 'intellectual property' (eg the amazon 1984 ebook scandal) but it seems to me knowing more about me, tracking and monitoring my habits and the like are vastly more valuable to outfits like amazon and apple than is providing some dead author's estate with 3 cents for every book sold, or seeing to it Bono or Mick don't end up hungry and on the street.

Michael Kassner
Michael Kassner

I, along with others have assumed the .govs have/use this ability. I have read of where with warrants, agencies used mobiles to record events. This is the first multi-sourced evidence that I have about commercial entities turning the microphone one. It is amazingly difficult for an individual to know whether the microphone is on or not, hardware-wise.

AnsuGisalas
AnsuGisalas

Base it off a computer-in-a-box with an OS you trust. Add a 3g modem. Add the phone software to a VM, allowing you complete control over how the phone software gets access to the physical assets of the machine. Add the functionalities with bluetooth or other such, mic and earpiece for starters. Presto. Not only won't it be listening to you without your consent, it'll also be pretty cool and powerful.

pgit
pgit

The lead is flat sheeting about 1/4 inch think and cut into 50 lb squares a couple feet on a side. One person could stack or remove them as needed by hand. (gloved hand for sure) After I dissolved the flight dept nobody wanted the lead so I inherited it. I have a few of the spare parts we had in stock, for an aircraft that is being retired in droves and thus parts are a dime a dozen. I have a radar wave guide and antenna, some landing gear parts, brake assemblies (the brakes themselves serve as wind chimes atm) and 2 NiCad batteries. Those babies are dangerous. The cells are all isolated so no runaway meltdown possible. But one of those cells makes for a spectacular cigarette lighter. (safety goggles recommended) Then there's all the railroad hardware I got from my grandfather...

santeewelding
santeewelding

When you get old, you forget Latin noun versus adjective. Hook up with a copyreader.

mtndive
mtndive

but you had better include one for the camera, too.

pgit
pgit

There's a lot more than you'd imagine. Pretty clever, really, when you figure the bots have to fill out a form and submit a "valid" (temporarily viable at least) email address. I sort of collect the names these bots come up with, and sometimes the language used in the body of a spam is downright poetic. Where we can hesitantly give lectures on morality to our necromancer. Class action suit of polar bear gets stinking drunk, and freight train over avocado pit self-flagellates; however, pocket living with cargo bay host..for tabloid teach dolphin beyond. Burglar for oil filter find subtle faults with garbage can of burglar. When particle accelerator beyond cup is South American, insurance agent for steam engine boogie inside ribbon. I'd never come up with that.

pgit
pgit

There has never been a need for GPS to track a cell phone. Each cell tower has a unique ID, just triangulate using a time delay and there you are... literally. The news report said "so far" the iphone merely writes this log to a file. But I have read elsewhere "law enforcement" has hand held devices that can download the entire contents of your hand held device, wirelessly, and that they are using these things increasingly at every excuse they can muster. At the border, at the airport, at the train station, soon at highway check points?

Michael Kassner
Michael Kassner

Are doing that. The telco providers are interested in that information. At least that is what several engineers have told me.

Michael Kassner
Michael Kassner

they certainly are difficult questions. I'm thinking the bad guys are beside themselves with all this to play with.

Michael Kassner
Michael Kassner

For the researchers that are capable of finding out this type of information.

pgit
pgit

I hadn't heard of that before, thanks for the enlightenment. If nothing else at least the indomitable spirit still survives. There appears to be vendors in the US selling something related, though from the introduction it isn't really clear that there's close to consensus on what exactly OpenMoko is, that is these US (or any) vendors may be selling 'something like' OpenMoko. http://wiki.openmoko.org/wiki/Distributors

apotheon
apotheon

My point is that I think exactly half a brain is still too much these days.

AnsuGisalas
AnsuGisalas

I know, but I did say exactly half a brain, not at least half a brain ;)

apotheon
apotheon

For a while there, at least half a brain was common amongst geeks who chose the Mac over the wintel stack. Things have changed, though, with the iPhone; its flaws, many of which are actually problems with Apple's management of the device rather than with the device itself, are increasingly unignorable, and the bar for recognition of the stupidity of using one as anything but a toy is dropping precipitously.

AnsuGisalas
AnsuGisalas

I mean, having exactly half a brain is what I've come to expect of the Bitten Apple cultists... :p

apotheon
apotheon

That's more along the lines of what I was thinking, assuming you're talking about stuff like the OpenMoko project (though I'm not current on the status of that project's legality in the US) -- or, even more basic-level hands-on hackerish, the TuxPhone project (I'm not sure whether it's dead at this point). If you're talking about stuff like the Modu, though, they're not nearly as "modular" as one might think. If you're talking about something like the Atrix, that's even less modular. There's another approach to modularity for the iPhone that looks interesting, but probably not very hackable (you know how Apple engineering works; "keep the user out"). Typical for the company, decided to patent the design, which means nobody else gets to do the same thing. Bastards. "If you want this feature, you have to use our smartphone -- which you never will if you have half a brain."

AnsuGisalas
AnsuGisalas

Would be fun to see if they can be combined with other devices... hybridized.

apotheon
apotheon

> Or... you could just build your own cellphone. Oh, yeah, piece of cake. Oddly, I have been thinking about the feasibility of doing just that. So far, I have not come up with an approach that fits within my budget and expertise (the latter of which is even less impressive than the former, on this subject). Keep in mind that my requirements include being able to comfortably fit my cellphone in a pocket of my motorcycle jacket, having a physical keyboard, and probably some other stuff that I've forgotten at almost four in the morning.

pgit
pgit

I never got around to collecting locks, though I have a couple keys. I hate to admit it but I could have boxes full of the things, not 19th century but 1900-1940's vintage. My grandfather was way up in the New York Central, before he retired (1964) he would bring me things the road was getting rid of. Of course I was excited by all of it, and in a few cases I recall him asking me if I wanted more (a lot more) of these things. One such was a kerosene switch lamp. The railroad had acquired untold thousands over the years, his job entailed authorizing the acquisition and disposition of signaling and other items. As an aside, he'd dealt with a company called star lantern, put them on the map by awarding them a contract with the central. He'd traveled to the place on the Central in the 1930's on a line that was abandoned by 1940. Years later we moved to that town, and when he came to visit one day we happened to drive by the unassuming factory on the edge of town and he suddenly remembered the place. We stopped in, and although the place had changed hands several times, and no longer supplied railroads, the present owner knew who my grandfather was. He said without him there would be no star headlight and lantern co. It's an even smaller world... Anyway, grampa gave me this switch lamp and asked me if I wanted more of them. I blurted out "a hundred!" He said it could be arranged. But my father jumped in and asked me where I thought I was going to store 100 lamps... certainly not in his house! My dad was uncharacteristically short sited when it came to railroad stuff. So I still have just the one. I also have the prototype of the Adlake model 400 hand lantern, given to my grandfather by the manufacturer for evaluation. It's mint out-of-the-box condition, despite my having used it extensively. All told my collection is best measured in the tons. The majority of it, in terms of sheer mass, lives outdoors. I would take your friend behind the woodshed for "fixing" those locks! They absolutely have more REAL value with bullet holes in them. I wouldn't doubt that the vast majority of such damage occurred during the depression. The original history of them is probably lost, but I'll tell you I'd rather 'invent' that history rather than "fix" the lock! ;)

Michael Kassner
Michael Kassner

My father was at the tail end of the war and was finishing his training when the surrender came. I remember that film. Talk about a small world. A famous and rich local businessman has a huge hanger full of airplanes. One of them is a Doolittle bomber. He refurbished it and it got re-certified on the Enterprise in the 80's. I don't think he still has it. Here is the link: http://www.goldenwingsmuseum.com/

santeewelding
santeewelding

A long-time, older customer collects and restores them. With my help, some become near-perfect forgeries, including the few which had bullet holes in them. I've always told him to leave those as-is for their charm (strongboxes, highway men, and imagination). But, he insists... His perennial complaint is that locks at his price range are getting harder and harder to find. Some rare and authentic, unretouched types fetch upwards of $30-40,000; much higher still for others. With our combined knowledge and experience we have been able to identify a few outright fakes. As with IT, that community has its underground, too.

pgit
pgit

I have a lot of heavy stuff, like an 800 pound, 12 foot tall switch stand. (and the lantern that goes on it) I have cast iron whistle signs, some mileage markers and a target signal that's about 4 feet in diameter. (and tons of little stuff) Shipping costs would be a nightmare. I'm envious of your friend, the north west is expanding passenger rail consistently. Here in western New York we used to have a rail network that was the envy of the world. I was born 60 years too late for my tastes. The reason I went into aviation was because rail was dead.

pgit
pgit

My father in law flew TBF/TBMs off of the Intrepid in WWII. He was the last pilot to take off before a kamikaze slammed into the flight deck. They ended up having to ditch and spend the night in the water. He said that experience was by far the worst of the war for him and his men. There is actually film footage of that attack. You see him take off, then the next footage picks up moments after the impact. I'm not sure of the status of the project, but an Australian film maker interviewed 'dad' for a history of the Intrepid. So what aircraft did your father fly? Sounds like he's still flying? My father in law gave it up suddenly, after some kid practicing his flagging sent him on a go around in a Corsair he was training in after the war. The torque of that huge engine at low airspeed put the left wing on the ground. If he'd been a few feet higher he said it would have killed him. He chased the flagman down and asked why he signaled go around, he said he was just practicing and randomly picked a plane to send off. He sank one Japanese supply ship and helped in sinking a destroyer, garnered a Navy Cross and survived countless flak, zeros and anti-aircraft fire... the irony of being taken out by a bored kid wasn't lost on him, he resigned from his reserve unit shortly thereafter. (but not before getting in trouble for buzzing the hospital where his fianc??e worked)

apotheon
apotheon

I know someone in northern Oregon who might be interested in some of that railroad gear, if you happen to be anywhere near him.

Michael Kassner
Michael Kassner

Lead is considered hazardous waste now, isn't it? What kind of railroad stuff do you have? My father (WW2 Navy pilot) is helping another friend (also Navy pilot) rebuild a second vintage airplane. So they have a hanger in similar circumstances.

Michael Kassner
Michael Kassner

I believe Chad is correct with populace. And, my use of populous was proof positive of Santee's advice.

apotheon
apotheon

You are aware the word he actually wanted was "populace" -- right? Speak English, man.

HAL 9000
HAL 9000

Are taped on the inside of the Air Lock Wall. It's [b]Bits of Body Parts[/b] that is causing that to stick there for the time being at least. Have you even seen just how small an opening you can get a body to go through when you have [b]Extremely Low Pressure[/b] on the other side. :^0 :D :^0 :D Col 0:-)

AnsuGisalas
AnsuGisalas

but some wiseass taped the instructions on the wall inside the airlock... no way I'm going in there...

seanferd
seanferd

He'll start turning the indicator light off and get a mirrored filter for his lens.

pgit
pgit

Too bad the 'red eye' isn't a necessary reality of web cam technology, eh? Creepy, but you'd sure know when you're being ogled.

apotheon
apotheon

It's probably pretty easy to write a script to do that for you on an Android device using SL4A.

AnsuGisalas
AnsuGisalas

That'd be nice otherwise too. Of course they don't like making anything that drains the cell, but hell, I want to know when my phone is eavesdropping, dammit.