Hardware optimize

Poll: Will your computer respond to a natural user interface?

Microsoft CEO Steve Ballmer suggests that 2010 will be the year natural user interfaces become a viable alternative to the mouse and keyboard.

In an article published on The Huffington Post Web site, Steve Ballmer suggests that 2010 will be the year of the "natural user interface."

But I believe we will look back on 2010 as the year we expanded beyond the mouse and keyboard and started incorporating more natural forms of interaction such as touch, speech, gestures, handwriting, and vision -- what computer scientists call the "NUI" or natural user interface. This process is already well underway through the proliferation of new touch screen phones and PCs, and in our growing reliance on voice-controlled in-car technology for communications, navigation, and entertainment.

Now, as is my way, I am skeptical that there will be a mass migration to a natural user interface this year or in the near future. There are still many kinks to work out with these interfaces. Making the same hand gesture three times just to get the page to scroll is not a good user experience. It is one I may accept on my smart phone, but not on my desktop.

What do you think? Are you ready to give up the mouse and keyboard for a natural user interface based on touch screens and hand gestures? Do you think natural user interfaces are ready to become the primary way we interact with computers?

About

Mark Kaelin is a CBS Interactive Senior Editor for TechRepublic. He is the host for the Microsoft Windows and Office blog, the Google in the Enterprise blog, the Five Apps blog and the Big Data Analytics blog.

46 comments
thomas
thomas

I have been typing for 28 years. I am good at it. Maybe for those to lazy to learn to type, they will look for some new device to save them from the chore. It may be ok for reading content. But I just can't see alternate content creation methods becoming better than keyboards for those who know what they are doing.

dogknees
dogknees

To see how an IDE is driven by "natural interfaces", or Photoshop or Autocad, or Access in design mode, or.... Once again we get categorical statements that don't mean anything.

Dr_Zinj
Dr_Zinj

1. Natural user interfaces that rely on gesturing for command input are far more physically intensive than mouse and keyboard. 2. Eye tracking doesn't work well because it induces twitching. 3. Verbal input and open office are mutually exclusive. 4. Using a natural user interface instead of a mouse & keyboard is like trying to make blueprints with fingerpainting instead of straight edges and sharp pencils.

rquance
rquance

Are we stupid or just playing the part of an idiot. A touch screen is NEVER as good as a mouse or keyboard. Filthy hands and many people never washing their hands is the number one reason I never use touch screens. I am not a germaphobe but it is so much easier to clean a keyboard and mouse than to be installing self cleaning devices on a touch screen device.

Gabby22
Gabby22

One of the problems with touch screens is that they're quite tiring, particularly if the screen is raised somewhat. This was discovered by research in the 70s and 80s and reduced the takeup of lightpens and touch screens. The original PC had a lightpen interface. Touch is fine for casual use (kiosks) and hand held (phones and readers), but not for hours on the job. Also if you don't use a stylus, the pointing resolution is much coarser than a mouse pointer. If you use a styler it's less natural.

CharlieSpencer
CharlieSpencer

Touch screens work very well in kiosk, point of sale, and other public environments. As to cleaning, if you compare manually cleaning a mouse and keyboard to MANUALLY cleaning a screen, the screen wins every time. No crevices for cookie-crumbs to fall into. No keys to fall off. In a public setting, no mouse or keyboard to get broken, stolen, or vandalized.

Forum Surfer
Forum Surfer

Look at your friendly neighborhood Walmart or CVS and you'll see that pesky redbox that so successful hollywood went after it and it's low rental prices. Then there's Bill Nuti's tenure at NCR after Marc Hurd released the reigns. Bill Nuti boosted NCR's growth and profit margins simply by expanding usage of touch screens and self service kiosks to turn the company into a self service giant. They basically brainstormed ways to utilize touch screen tech in self service industries in a more widespread fashion.

CharlieSpencer
CharlieSpencer

at most big box retailers. Options to scan or pull from assorted media. Options to print in various sizes or save to CDs. Options to create assorted 'hard' products like calendars, mouse pads, T-shirts, greeting cards. All touch-driven. Ever check in at the airport? The whole process is touch driven. The technology has it's place.

detours
detours

I will be tough to convince of the benefits of so a called 'natural interface' to the computer, given the current crop of alternatives. I can see huge benefits to the disabled, but otherwise only limited applications. Throwing out my mouse and keyboard will only slow my work down and make me less efficient. Text to speech vs reading a. I can read email and documents much faster than a computer can read them to me. b. The computer can't read or describe pictures or video c. Environmental noise, like radio, TV, kids, coworkers and other applications using text to speech will interfere with my ability to hear what my computer is telling me. Speech to text vs typing: a. I can type a document almost as fast as I can speak it. But corrections, revisions, insertions and formatting by speech are very complicated and take much longer by voice. b. Multitasking: With a keyboard, I can switch to background programs, do something and switch back to the first one in a flash. With speech to text, I have to exit document input, use voice commands to call up the other program, tell it what I want to do, correct errors, switch back and re-enter speech to text mode. c. Speech to text is ruined by environmental noise like radio, TV, phone, kids, coworker conversations. It requires a quiet environment, tries to type coughs, sneezes and interjections, and doesn't let you think out loud. d. Text to speech for scripting, coding and SQL queries would be a nightmare. Touchscreen vs keyboard & mouse: a. Cursor movement - a mouse can move the cursor across my 23" monitor in a few inches. Similarly, keyboard shortcuts allow me to select and format with only a few keystrokes and only a couple of inches of movement. On a touchscreen, my finger has to cross the actual width and height to move the cursor. b. Precision - a mouse has pixel precision. Fingers are too fat for many buttons, links and checkboxes and precision tasks like picture editing and even text selection on a touchscreen. c. Multitasking - a mouse has multiple buttons for programmable tasks and can incorporate gestures. Fingers can also use gestures, pinching and zones but a tap is still just a tap (not programmable). d. Ergonomics - There is risk of repetitive stress injury. But I can still create an ergonomic workspace with a mouse and keyboard. With a touchscreen, I have to hold my arm in the air. Installing it into a table will strain my neck and back leaning over it. e. Speed - touchscreen keyboards will never be as fast or accurate as a real keyboard or 10-key interface because of dropped characters, calibration issues, parallax and lack of tactile feedback. Glove sensors: Tracking finger and hand movement is cool, but I can't see a speed or accuracy improvement when you remove tactile feedback. Tactile feedback is an important part of a human being's 'natural' interface with the world. 'Traditional interfaces', like mouse, trackball, trackpad and keyboard, give us natural feedback that our muscles and senses can learn. Any so-called 'natural' user interface that removes tactile feedback is 'unnatural'.

Forum Surfer
Forum Surfer

Try using man sized fingers to write a text message on a tinier than average touch screen phone. I could only imagine the frustration of me trying to click a single check box on my monitor while my finger continues to hit 3 check boxes at once.

zclayton2
zclayton2

and anything that can interpret my ax-murderer scribbling at more than 1/3 the rate of my typing will be challenged for quite some time. It will have to: read intermixed print and script, figure out the 4 or 5 different shapes of n, and other variable shape letters, figure out in context the letters u-n-m-w, l-b d-cl, l-b-k, i-e-c, y-j, g-cy, s-r, k-h, and other issues I can't think of off the top of my head. I will type. And my kids have asked why they have to learn script AND printing since they keyboard everything anyway.

mwkeeley
mwkeeley

The take up of natural user interfaces will depend on how effective they are, how much benefit they yield to the user and how cost effective they are. There is no point in changing to a technology that is not 100% effective, however if it provides substantial benefits by all means use it. I use touch screen, handwriting recognition, text to speech conversion very effectively in my daily routine, but I still use the trusty mouse and keyboard.

CharlieSpencer
CharlieSpencer

"There is no point in changing to a technology that is not 100% effective,..." By that philosophy, we'd all still be living in caves and cooking over open fires. Very few of our current technologies are 100% effective. Most are implemented because they're economically less costly than what they replaced. Light bulbs are grossly inefficient but they're less expensive than gas lamps and candles, especially when the fire potential is eliminated.

boggs
boggs

Light bulbs are "grossly inefficient" as compared to what? Luminoso! Also a previous poster says natural interface would reek havoc with hunt and peck users. I believe that is who such devices are geared toward. If you don't type, a keyboard is quite clumsy. The letters aren't even in alphabetical order, hence the need to hunt before the peck.

Forum Surfer
Forum Surfer

That had been converted to DVORAK. Very few people ever used them, but I remembered a few. Back in those days, you didn't toss out a perfectly good keyboard. If I had to relocate that thing, I had to move all the freakin keycaps. I still have a handful of Model M's. My kids learned to type on them. I feel their tactile nature is beneficial to their learning to type. Plus they think it just sounds cool as you learn to type faster. But man, back in the day if you were in a room full of Model M's and data entry people...man what a commotion! I find it funny that one of the high end keyboards for gamers and uber geeks is essentially a rewired Model M: http://www.daskeyboard.com/

CharlieSpencer
CharlieSpencer

I was not referring to all natural interfaces. I was responding to only to Forum Surfer's comments on visual interfaces. 'Hunt and peck' users look at the keyboard more than the screen. If one is not looking at the screen or other visual interface, how does one use it as an input? Light bulbs are grossly inefficient in converting electricity to light, expending over 70% of the energy as heat. Yes, the conventional QWERTY layout is clumsy (deliberately; it slowed early typists so they wouldn't jam the hammers), but the Dvorak layout is far more efficient. A single 12 to 16 week typing class in high school can make users of either layout more efficient than 'hunt and peck', and would benefit students more in the long run than another PowerPoint class.

doug
doug

The Querty keyboard has lived for a 100 years; need I say more . . .

john3347
john3347

Yep, it sure has. But the typewriter also lasted for many, many years, too; need I say more? Maybe 10 to 25 years from now, the qwerty keyboard (and mouse) will be nothing more than a memory in the minds of the old farts.

TheProfessorDan
TheProfessorDan

but it will happen. I think that we are already heading in this direction. I think that there may be applications that will always require mice and keyboards such as word processors and spreadsheets. I can't see any of these "natural" interface devices work for word processing.

twitteman
twitteman

Not much to add to Dan's comment

NickNielsen
NickNielsen

When they get all the kinks worked out of the speech recognition software and it recognizes exactly what you're saying in spite of regional and national dialects, second language accents, and speech impediments. It will probably be cheaper (and easier) to perfect the direct cranial interface.

NickNielsen
NickNielsen

I can barely use my laptop without fat-fingering something; programming the GPS requires that I use a stylus of some sort. And I can just see the 1-or-2-inch fingertip gesture that works fine on my Zune or phone scaling up to 8-10 inches on the touch screen...at arm's length. I've already said it. The full-size touchscreen is ideal for certain apps (information kiosks and restaurant point-of-sale, for example), but not yet ready for prime time.

CharlieSpencer
CharlieSpencer

Well, people have been trying to get the bugs out of speech and handwriting for a while. Until we can standardize speech patterns and handwriting, these will remain niche applications. The keyboard remains the most effective way to enter text, and an even a minimally competent key pad user can enter data faster than speak or write it. Replacing the mouse may be easier. Vision of a sort is available using a forehead-mounted pointer, calibrated to point where the user is looking. So far it's use has been limited to those physically unable to use conventional mice or keyboards. I'm not sure gestures are. I also don't know about the cost of these devices or how they'll overcome some of the advantages of mice and keyboards. I can type in the dark; will the devices have infrared sensors? And like others, I can't abide fingerprints on my screen.

Forum Surfer
Forum Surfer

As an odd multitasker I'm often looking at a book, manual or another screen on a second pc while clicking and scrolling. I could foresee that wreaking havoc on my interface. Having spent hours getting feedback for developers and field testing restaurant pos applications, I can honestly say touch screen isn't for me. I hate all the fingerprints, as well. And good luck getting me to use one of these fancy interfaces on my favorite FPS game. I like my keyboard and mouse thank you very much! If I want some new human interface gizmo, I'll go play my kids' Wii. It has also taken me years to find a mouse I consider perfect, so I'm not willing to say all those years searching were in vain just yet!

CharlieSpencer
CharlieSpencer

The 'hunt and peck' crowd would really have problems. What mouse are you using? I'm a big fan of the Contour Per-fit. Four right-handed sizes, three for lefties for a better ergonomic fit. Three buttons, thumb scroll-wheel, and a two more thumb buttons.

Forum Surfer
Forum Surfer

That is a way to solve the problem I would have never tried. Glad it works for you though. For me, the switch to a hyper sensitive, DPI overkill, fingertip style laser mouse did the trick. For typical work tasks my arm stays stationary and I don't move outside of a 2-3 inch square on the mousepad. Gaming is different, but very seldom do I need to "reset" the mouse back to center of the pad. On a side note a member of Risk Management once asked for recommendations for ergonomic mice. All of these unique differences are just further backing up what i told her. My response is that every user has different needs and there is no one size fits all solution. I have one coworker that is in absolute pain if she uses an "ergonomic" split keyboard for more than 30 minutes due to a weird rotator cup issue. I have yet another coworker that swears her carpal tunnel lessened after using the exact same keyboard. To each his own...

CharlieSpencer
CharlieSpencer

I find I move my mouse laterally at the wrist, and at the elbow or shoulder for up and down. I was starting to get CT in my right wrist several years ago. I moved my mouse to the left side and changed the buttons. Now my primary work computer is configured southpaw, while my home computer is still right-handed (for my wife). This splits the work and reduces the 'wear and tear'. It took about two months for me to get use to the change.

Forum Surfer
Forum Surfer

I have beginning stages of arthritis developing in my mouse hand. It isn't in the joints, however. It is in the area where I broke all the metacarpal shafts on that hand (motorcycle wreck at 21). I'm sure this is a factor in my preference. Maybe that's why "ergonomic" mice that you move via your palm actually cause me pain? I just find it odd that a non-ergonomic, ambidextrous mouse came to be the most comfortable for my use.

john3347
john3347

I am arthritic and find a thumb operated trackball is MUCH more comfortable and easier to use than any mouse I have ever used. I also have to rely on third party software to program the center button (scroll wheel) to perform a double click because I have difficulty coordinating two rapid index finger motions for efficient double clicking. This just illustrates the differences in needs among users. As different natural user interfaces are developed and become widely available, I will try them and if they improve my "computing experience" I will embrace them. If they prove worthless to my needs, I will reject them. This is just how the world turns. As a prediction, I believe that while there will be progress in the development of this natural user interface hardware and software during 2010, I don't see alternatives to keyboard and mouse (or trackball) becoming mainstream in 2010.

Forum Surfer
Forum Surfer

I'm told my hands are medium to large. I looked at the puny, flat copperhead and thought it was far too small, light and flat even. Most of the time I have my thumb on one of the rubber like ridges that jut out, next two fingers on the buttons, ring finger grasping the other side and my pinky kind of floats or rests on the pad along with my palm. Sounds ridiculously uncomfortable, but not so in practice. Like I said, given a few weeks I grew to love it. I have discovered, however that I have odd tastes and I am almost always the odd ball in terms of preference. Back in the day I loved trackballs, but once I spent more and more time at a desk I grew to hate them. Razers have some bad reviews on the net. That is usually caused by users who don't follow the very specific firmware updating instructions. Like anything else, make sure you get the latest firmware and driver uploads from the manufacturer...not the outdated cd that comes with. Other than that, side scrolling, price and the fact that the blue lights are "too geeky" at work are the only negative points for me.

CharlieSpencer
CharlieSpencer

I'd love to have the settings on-board, but it may be too narrow for me. The specs list it at 66mm; my Per-fit is over 90. On the other hand, it's $40 less. Maybe when the tax return gets here... Regarding the Per-fit's size, it comes in seven sizes. While it works great with XP, the fourth and fifth buttons for the thumb don't work correctly with Vista or W7. I've been nagging them for a couple of years to update the drivers, but all I get is the cold shoulder. I stick with it strictly for the ergonomics of it. Maybe public embarrassing them will help? :D I tried a 3M 'vertical mouse' once. It had a flat base that housed the LED with a vertical handle you gripped like a joystick. It had a two-position rocker on the top for left- and right-click. It looked great, but resting the weight of my hand on the pinky-side against the base quickly became tiring. Hands are heavier than I thought. I hate a trackball. They give me thumb or wrist pains faster than anything else. I get cramps in less than 30 minutes.

Forum Surfer
Forum Surfer

I'm far too clumsy and I kept inadvertently hitting them far too often. I used this one by accident when working on a relative's son's pc. He was one of those 24/7 gamer types and I was surprised I had to be called in to fix it. But he had a Razer Copperhead. I have since bought one for home and work. It's just the right size for me and more of a fingertip mouse than a palm mouse. I find this funny because I always [i]thought[/i] that I preferred palm oriented, ergonomic mice. After giving up my ergonomic mouse and using this guy for two weeks, I no longer suffered from occasional wrist pain. If you like extra buttons, this may not be for you. It has two extra buttons on each side, yet the mouse is symmetrical. That makes it great for lefties, but those two buttons on the far side seem awkward for my pinky. Then again, any pinky oriented mouse buttons seem weird for me. I love that the settings are stored in firmware on the mouse. If I move it, my custom settings stay with the mouse! I've come to love it's fingertip style light weight and hyper sensitivity. I don't foresee myself ever returning to ergonomic, heavyweight palm oriented mice. I like the rubber-like texture that doesn't seem to rub off after a few years of use and doesn't look icky after a few hours of palm sweat. I hate that it is a tad pricey. I find it very odd that after all these different mice, the one I like so much (so much that I bought a second for work) has precious few extra buttons and little to no redeeming ergonomic features. For what it is worth, I'm one of the most frugal people I know, so buying a 2nd model for work is really saying something. My 6 year old son has even learned by example with my frugality by doing google video searches for transformer (or GI Joe or whatever) toy reviews before buying one "to make sure he gets a good one." Wow, had to google that Contour Per-fit and I must say, that is a huge mouse! That looks like something I would have tried back when I craved extra buttons and wheels. My GIS people seem to love the Logitech Performance Mouse MX and it's free spinning scroll wheel. Helps on scrolling large distances and huge spreadsheets if one was to ever do such. I found it novel, but not something I could use on a regular basis.

Mark W. Kaelin
Mark W. Kaelin

What do you think? Are you ready to give up the mouse and keyboard for a natural user interface based on touch screens and hand gestures? Do you think natural user interfaces are ready to become the primary way we interact with computers?

TtfnJohn
TtfnJohn

Now, let's see in reality how it might work or not. As a Canadian living in coastal British Columbia I use words every day that puzzle the daylights out of folks who live on the eastern side of the Rocky Mountains. Perhaps Ballmer does too, cause I'm perfectly understood in the Puget Sound area of Washington State (or Warshington, as they pronounce it) while people in Spokane look at me as if I've suddenly taken leave of my senses. Of course there are the dialects in the southern United States, accents and dialects all over the British Isles, Jamacian english, different african english expressions, Indian english and need I go on? Gestures? Will some PLEASE tell me what the hell a gesture is and why in heaven I ought to use one? To me they all seem like wave the mouse madly and pray. Handwriting? Is the McLean Elementary school course in handwriting about to reappear along with the ruler to ensure you held the pencil or pen in exactly the right fashion in exactly and only the right hand and copied beautiful soulless script? Just exactly how? Scanners have enough problems as it is with printed text! Touch? So many fingerprints that I can't see the screen for them? Vision? New goggles or pointers attached to the forehead? Oh, please! I'm sure I'm supposed to use that while driving 110 km/h (70 mph) down the freeway, right? At some point maybe. Soon? Nope. As it stands right now with some apps I'm faster with keyboard and shortcuts along with a little mousing than I am any other way. With others, say Photoshop, where it makes sense to use an alternate pointing device I use it. The last thing I need is the big computer god in the sky to tell me how to speak my languagę, how to do cursive writing on those increasingly rare times I use it, how to wiggle my fingertip at a screen or other silliness. The research will come up with things that are worthwhile and useful. As far as a general purpose UI, not yet. Perhaps, given all the variables, not ever. ttfn John Beam me outa here, Scotty!

wesley.johnson
wesley.johnson

When your pointer finger wears out from mousing, you will happy to have other options. Handicap people already use VOICE. Touch Screens abound, no mouse at my ATM or Notebook.

CharlieSpencer
CharlieSpencer

for specialized or single-purpose systems. You don't find them on many general purpose Windows / Mac / Linux systems.

jck
jck

I'm ready to give them up for the most part. I was hoping by this time, we'd have that Star Trek tech where just talk to it in plain language, and it understands. But instead, we virtually have what Scotty had...and we are still talking into the mouse. "Oh computer????" :^0

yakupm
yakupm

Many years ago (20 or so?), in the pre-mouse days HP came out with the 150 which had a monitor overlay that intersecting beams of light. When you 'touched' the monitor, breaking intersecting beams, it would respond. Then the mouse came along and no more finger smudges on the screen. Personally, I find the mouse a hassle as it forces you to remove your hands from the keyboard so I am a BIG keyboard shortcut fan. Touching the screen is NOT the way to go, it will be intersting to see what develops but don't excpect the keyboard to disappear for a LONG time.

dialfred
dialfred

1. Voice - No- it's hard on the throat & it can make a room full of workers sound like a bunch of telemarketers. I like quiet when I work. 2. touch screen - OK for a bit, but holding my arm out all day to touch a screen is more tiring than typing at a comfortable level. 3. hand gestures - same thing, after some time my arm will be shaky at best. They do have a place, but will work better when the touch panel is in the desk.

gcdimarketing
gcdimarketing

I am waiting for the plug & play version of natural user interface: Where they install a wireless connection directly in my head so I can connect with and operate my devices by thought.

Ohsolost
Ohsolost

...is not at all practical. Imagine overhearing your coworker's every command to open a file or application! And the ergonomics for touchscreen...for me the ideal position of my monitor is a few inches out of arm's reach. And the fingerprints...ugh! Touchscreen is great for some specific applications, but not for most office computing environments.

martian
martian

What about the overhead of getting it to recognize all the quirky ways different groups use to express themselves... Think teen or redneck, take your pick... ;)

stuartc
stuartc

Me as a developer, will never be able to sit with arms outstreched for 8 to 10 hours a day. Great for building the arms and shoulders, but thats why I go to gym. I use a HTC touch phone and i can even use a normal (non touch) phone any more. There the touch works beautifully.

Peconet Tietokoneet
Peconet Tietokoneet

Well i have not seen how this is going to work on a server, so i will say "live on the keyboard and mouse". I can just see it now dirty, or, chocolate fingers going all over the screen... yuk. No thanks. I will stick to my faithful keyboard and mouse.

usamare
usamare

Trying to get people to use split keyboards and marble-mice so they won't get carpal tunnel is like pulling molars. Gestures is just right out! Added cost plus the user has to learn something? I'm sorry but I just don't see this happening with the bulk of the current (logic defying) user base.