Hardware

The mouse mentality holds down mobile apps

Because we have become so accustomed to writing user interfaces that use a mouse, it is difficult to write applications for mobile platforms with effective user interfaces. Justin James explores what needs to change for mobile applications to have usable interfaces.

The mouse defines, in large part, how we compute -- which is great for most needs. Having that "common ground" in terms of the interface is wonderful for probably 90 percent of the applications out there. The problem is the other 10 percent of applications that would be much better with alternative input devices. Even more problematic is the increasing popularity of devices with alternative input systems. Many UMPCs and smartphones use a touch screen with a stylus, and/or a cross pad for pointing. Neither of these devices are good for drag/drop operations, and a cross pad is particularly bad at complex maneuvers (other than the "Konami Code" of course).

The realm of video games is quite instructive to us. You see, there are certain game genres which simply are much more popular on consoles than on PCs and vice versa. In fact, some genres of video games barely exist on one platform or the other. The general rule of thumb is that video games that require precise, rapid motion tend to do much better on the PC. To be successful on consoles, for example, first person shooter (FPS) games need much less precise aiming than they do on PCs.

Likewise, the real time strategy (RTS) games do not exist at all on consoles for all intents and purposes. The holdup is not necessarily the keyboard; modern console controllers are just as good for moving in a RTS, and I personally never use the keyboard in an RTS except to pause the game, and I do just fine.

The problem is the age-old mouse vs. controller issue. In fact, people who play a lot of console-style games (including using a console emulator) tend to purchase console style controllers for their PCs because the keyboard and mouse is a poor choice for games designed for controllers.

This is a pertinent observation because an awful lot of those UMPCs use console controller style cross pads as their pointer when held in a normal position. Cell phones and smartphones use a cross pad as well. Some smartphones also (or only) have a tiny trackball, like the newer BlackBerries. I was worn out after using that trackball after more than a minute, and I highly doubt anyone can get much of anything done on them.

This means that the big market to push new applications into handles input in ways that are unfamiliar to the vast majority of application developers. Even worse, the video game connection shows pretty clearly that the interfaces (and even the types of work that we are used to performing on PCs) is extraordinarily difficult in an environment with no mouse or a mouse alternative. This means that, for the mobile form factors to make their predicted impact, at least one of three things needs to happen:

  • We dramatically redefine the way we think about user input. The changes made to FPS games show us the way; making pointer targets "magnetic" to draw the pointer in and requiring less pointer precision will lead to success. This happens at the OS level.
  • We redefine the applications completely that we expect people to use. There is a reason why RTS games on consoles almost never succeed: The only input device that works well for "all over the screen" rapid input happens to be a mouse. A stylus might be just as good -- I would need to ask a Nintendo DS owner or three about that. For applications that require this type of input, we need to think completely differently about them, period.
  • We design devices and applications in a manner that the devices can be docked to a mouse, keyboard, and monitor, and when docked, the applications seamlessly "unfold" from an interface well suited to the mobile device to a more traditional desktop computing interface.

I think that the last choice would be brilliant, but it requires a lot of changes in the way device makers, OS coders, and application developers think about interfaces. Until we are no longer in a mouse-centric mindset, we will not be able to write applications that will be truly useful and usable on mobile platforms.

J.Ja

About

Justin James is the Lead Architect for Conigent.

23 comments
JosB
JosB

In my opinion the device that comes closest to the required functionality would be a grapical tablet like the ones Wacom produces. The difference between the stylus of a smartphone and the stylus of such a tablet is that it has working buttons and is highly programmable. Even the type of motion can be programmed to a feature (example: select + fast up for copy, then fast down for paste). The stylus does not has its own powersource which helps keep the size down. The only huge disadvantage of a tablet is that pure keyboard input is much faster than anything you can do with handwriting when dealing with text. However, for mobile devices this is less of a problem since you don't have a full-size keyboard anyway. From my experience, if you can minimize the amount of text the users need to input you should be able to replace the keyboard/mouse combination with a intelligent stylus mechanism.

Justin James
Justin James

I didn't realize that people use tablets for that many kinds of input. Thanks! I may try to see if I can get one inexpensively and check it out. J.Ja

JosB
JosB

Justin, While a inexpensive tablet might indeed be a good test you might want to find out if there is someone you know who has a professional A5/A4 or larger tablet using Vista or a Mac. If you know someone, see if you can use their tablet (even on their computer) for about half an hour to one hour doing day to day stuff. The reason I surgest this is that it's easy to use the smaller (and thus inexpensive) as replacement for the mouse. A larger tablet takes a lot of space and 'forces' you to do as much as you can with it. Frequent swapping keyboard and tablet is very annoying and tablets do have disadvantages. I think you will experience both the pros and the cons better with a larger tablet but those are too expensive for a test.

Justin James
Justin James

That's a good suggestion, but unfortunately, I do not know any tablet users. I might see if I can find one inexpensively on eBay, like I did for my keyboard (a specialized KB that are pricey new, but I got one in like-new condition for about 30% off with some patience). J.Ja

priyas
priyas

Whenever i use a mouse my body parts are comfortably positioned. Relaxed chair for the back, stable ground for the mouse, considerable sized canvas to view in the form of the monitor. When I use a mobile, this comfort is absent. Probably if i had a larger screen ehich is collapsible.....

Justin James
Justin James

That is a good point... desktops are usually in a workspace customized over time for a user's preference. Mobile devices are used in ad hoc workspaces most of the time. That makes a big difference, and mobile interfaces need to continue to improve and take that into account. J.Ja

C_Tharp
C_Tharp

The article highlights the limitations of input devices and puts forth a few ideas for changing the existing solutions. Surely, ideas must be plentiful in the reader community. My idea is to change the way that touch pads operate. Touch pads are programmed to operate as though it is a different kind of mouse. A mouse moves a pointer on the screen. All motion begins with the current position of the pointer. Moving the pointer to a distant place with the touch pad often requires a lot of hand motion. A touch pad could be more like the touch screen part of a touch screen terminal. The pointer could be instantly moved to the point where the touch occurs. Sliding the finger across the pad would move the pointer as you would expect. For a distant move, just pick up your finger and touch somewhere else. It lacks the precise connection of the picture to the touch screen. The user must associate the two through eye to hand coordination that is offset from one another (the screen is in one place and the touch pad in another). It also lacks the precision of a touch screen that is as large as the terminal screen. The idea is to use an existing device in a different manner to provide a faster and more direct response. This is, of course, a cheap alternative to a touch screen terminal.

Justin James
Justin James

I agree, touch pads in their current implementation are horrid. I disable them on laptops, and use the stick when I can't plug a mouse in, and plug a mouse in whenever possible. I like your suggested alternative, and I am sure there are others out their too. But to try to use a touchpad to replace a mouse... disastrous. J.Ja

mosw
mosw

I don't like current touch pads but I think with some changes they could become a unifying pointing device. For mobile devices I saw a demonstration with the touchpad(s) on the back. You hold the device in one or two hands and move you fingers around on the back to control the pointer(s). This has two advantages that I can see. 1-More precession, you finger isn't blocking the screen so you can have a precise pointer-target. 2-The range of motion needed can be small, so you don't have to move your entire hand, just your fingers. To me this is much better then current touch screen approach. On notebooks/desktops extensive adoption of multitouch capabilities, like Apple is doing, might get me to use a touch pad. Once you introduce multiple cursors and multi finger gestures a whole new word of interface interaction opens up. My 2 cents.

Justin James
Justin James

I saw the same demos, Microsoft or Compaq, if I recall. Awesome stuff. This is the kind of stuff that will make touch screens really useful. Right now, touch screens are really inaccurate and really just function like frustrating mice. I think allowing them to function less like mice (attempting to emulate precision pointing devices) and allow them to function more like GUI widget manipulation devices is the way to go (rotating and on screen dial, "squeezing" something to scale it, etc.). The mouse can be viewed as a physical object to be manipulated by the hand to create a precise effect, new touch screen technologies can and should be viewed as software mice, not hardware mice alternatives! J.Ja J.Ja

myrdhrin
myrdhrin

Justin, This applies to more than just mobile platforms... I work in the document industry and users rely almost solely on the keyboard for efficiency when correction data on imaged documents. Every time you bring someone new on board (testers, developper, anyone) you have to pull the mouse away from them otherwise they think, develop, test the application in a way that will be foreign to the end user. I guess it all comes down to knowing your user *grin*

Justin James
Justin James

Knowing the user is right! Not just knowing the user, but knowing how they do their job. That's why I like to ask "why do you do this?" not "how do you do this?" when designing software. When you ask "how", the user tells you the broken process that their manager tells them they should be using... if it is so great of a process, why ask me to write it as software? When you ask "why?", you find out the true goals. Observing the user in their natural habitat ("knowing the user") is the best way to find out what is good, and what is not, about the "how", to be used as a guideline for your project. Whenever the manager (project sponsor) tells me to do something dumb, I can point to the observations and say, "look, you are already doing it that way, and it stinks" or "the users are doing great with a different way, so why does it need to change?" Failure to heed how the users actually work results in "shelf-dust-ware", not "software". J.Ja

aikimark
aikimark

I teach my students that any application design that requires the user to move a hand from keyboard to mouse (and back) is a bad design or implementation. ALWAYS provide for keyboard navigation and mouse-less operation.

Justin James
Justin James

The movement to put a hand on the mouse is extraordinarily inefficient, and completely breaks the workflow, there are a ton of studies to support you on that too. Apps should ahve their workflows split into "mouse parts" and "keyboard parts" to minimize this, and users need to be better eduated to understand that you can navigate between fields of input without the mouse! And automatically moving focus is NOT the answer, either. That just confuses users. :( J.Ja

srivera
srivera

Seems like everything is geared toward the mouse. I love my keyboard and keyboard shortcuts. Mouse has its place, but for any degree of accuracy or speed, give me the keyboard. Every software developer should have mandatory heavy data entry where accuracy is critical before ever being cut loose on developing software.

Justin James
Justin James

When I was in 8th grade, I took a CAD course, and everything was keybard input, no mice. It was like that, typing in a vector, telling it to remove the vector (as far as I can remember, it was easier to re-create than to edit existing items), and so on. Was it frustrating? Not really, certainly no more than using an image editor with a mouse. And in some cases, it gave a level of precision which is quite hard to get with a mouse. It was in the gross stuff (select a huge part of the screen, for example), where it didn't work as well. But at the time, I had never worked with a mouse either, so it did not feel unusual to me. I suspect that if I were to go backa nd try it again, I'd be quite frustrated by the lack of a mouse. J.Ja

mattohare
mattohare

You know I do almost all of this by keyboard? I might drag a few items onto the surface. When the form gets crowded though, I use the keyboard to set the size and location properties. Back when I could do it from an Immediate (debug) Window, I'd set all of those properties there at the command line and watch all the controls move into position.

Justin James
Justin James

... is watching bank tellers, airline ticket counter personell, or other data-entry heavy people work with their PCs. It is almost all keypad + arrows, with a bit of typing, no mouse. And they are extraordinarily fast. The general rule of thumb is that keyboard input is faster for the experienced, trained professional using software that has been designed to be keyboard driven. Mouse/pointer input is faster for the casual user, or in situations where manipulation (as opposed to put input) of objects is required. For example, imagine trying to use Photoshop with only a keyboard, by typing in coordinates for everything. Pretty miserble, eh? In other words, keyboards are perfect for some tasks... but for other tasks, they actually make things worse! J.Ja

john
john

well as for the nintendo DS the stylus is useful although can become cumbersome having watched my children and numerous other children play these devices, hence they have something of a keypad attached to them. i tend to agree with you that we need to think of ways to obviate our way from the mouse, but where does it leave all the people that either dont have the nouse to learn or even want to change habits formed in the windows minds-set. perhaps we are heading for the finger as a replacement to the arthretic inducing device that we loving cling to, with the next generation already on display from Microsoft, the rest of the computing world will need to play catch up (unfortunately) will keep looking back for further input into the future of computing! ^_^

Michael Kassner
Michael Kassner

Great job Justin, yet again something that a network engineer does not think about. I do now realize that to mouse or not to mouse is actually a big deal.

Justin James
Justin James

Michael - Nice to know you liked it! I am also happy to see that my writing is not inaccesable to non-programmers. :) The mouse has dominated input for roughly 15 years now, basically during the Windows 3.X era when everyone ended up with Windows (or X Window on *Nix). And it has been great. But the wave of mobile computing devices is forcing folks to deal with non-mouse input devices, and it is a huge challenge to developers to step up to the plate. The best comparison is what WiFi networks did to security, in terms of radically changing the playing field, with a ton of tradeoffs involved and an unquenchable thirst for the new stuff from the users, fueled by what they had at home. J.Ja

mattohare
mattohare

I liked your article, except it seems another plank on the pointer-device-only platform. Since I?m from the command prompt days (VMS/DCL and DOS), it took me quite a while to get used to the idea of a mouse. Now of course there are areas where I?d prefer to use a mouse. That said, I do encumbered by a lot of applications that seem to require it. I do a lot of my commuting on public transport (planes, trains, busses, ferries) and in non-traditional areas (cafes, parks, bus terminals, train depots, ok, you get the idea). Some of these places, an external mouse is very difficult, others, it?s downright impossible. (Ok, you try a mouse on the top deck of a Belfast Metro bus!) I?ve been lucky, until the notebook computer I bought last night, to have a pointing stick. Having the mouse function on the keyboard was a miracle for me to a level that I could never do justice with mere words. Finally, I could use a pointing device without removing my hand from the keyboard. I?m sad to see that such a helpful thing is so difficult to get now. You ever notice some people that seem to go overboard with their mice? To log in, I see people click in the username box and type their name, then click in the password box and type that, and finally click the login button. It?s a wonder to me that they can get anything done if that?s how they do it all day.

Justin James
Justin James

"I liked your article, except it seems another plank on the pointer-device-only platform." I think that pointers, or something along those lines, are here to stay. Haptic interfaces intrique me. My girlfriend and I just bought new cell phones, she ended up with the LG Venus. I thought I would hate the touch screen until I realized that it give mild feedbazck via a vibrate to indicate a "push" of something. Even a touch screen is a pointing device input, of course, but it is quite far removed from the mouse paradigm, and a stylus is somewhere in between. Pointing devices are pragmatic, whereas keyboards are symbolic. "CTRL+V" for "Paste" requires a distinct piece of learning to occur; dragging the selected item to the new location is more "intuitive". The way I think of it, keyboards do not work, or work poorly if the user is incapable of ever creating a new memory (think of the "Momento" guy). Pointing devices work for that user, albeit probably rather slowly. "That said, I do encumbered by a lot of applications that seem to require it. I do a lot of my commuting on public transport (planes, trains, busses, ferries) and in non-traditional areas (cafes, parks, bus terminals, train depots, ok, you get the idea). Some of these places, an external mouse is very difficult, others, it?s downright impossible. (Ok, you try a mouse on the top deck of a Belfast Metro bus!) " This is *precisely* why most laptops end up as "portable desktops" and not "mobile computing devices". Personally, I tired quite quickly of dragging the laptop AND the mouse to meetings, and I hate the "stick" and the touchpad, so I left it all behind and just brought a 99 cent steno pad with me everywhere, like I did in college. It's a perfect solution. "Finally, I could use a pointing device without removing my hand from the keyboard." Yup, this is a real efficiency item. The "right hand to mouse and back again" motion is a huge break in workflow, and makes it fairly impossible to use the two in conjunction except the way First Person Shooter games do. "You ever notice some people that seem to go overboard with their mice? To log in, I see people click in the username box and type their name, then click in the password box and type that, and finally click the login button. It?s a wonder to me that they can get anything done if that?s how they do it all day." I have noticed this too, it's because, unless someone shows you that "Tab" to get between fields is constant on the OS, you will NEVER find it out. That's what I mean by keyboards being symbolic. There is no button labled "Next field", and the "Shift" key is probably better off labeled "Invert" ("K" is "k" inverted, SHIFT+TAB does previous field, the "inversion" of "Next field", etc.). The keyboard contains nearly zero direct correlation between the keys and their purpose, other than the letter and number keys themselves for typing purposes. This is what makes it hard to learn, but at the same time, quite pwoerful when learned, thousands of commands all easily accessed and strung together at will, kind of like using pipes and redirection on a *Nix system. It also says a LOT about the applications, and the jobs people do, that this level of inefficiency is tolerated. Imagine waiting in a long line at the bank and seeing a teller working like this? They would need to double the security team, to protect the workers from angry customers. J.Ja

Editor's Picks