Someone interested in the history of of User Interface (UI) will also know that the history on that front has not been quite so dramatic. From the first popular consumer mouse in 1984 to the present day, very little has changed about how we use computers. Mouse and keyboard, windows and icons – up until about 2007, when people started to get excited about NUI.

What NUI is depends on who you ask. People like Microsoft’s Chief research and Strategy officer, think that NUI means doing away with the mouse and keyboard entirely. Instead we should be doing all our computing with a tap of the finger or a gesture of the hand. Not to mention Steve Jobs’ claim that if you have to include a pen in an interface, you’ve screwed up.

While this enthusiasm for futuristic interfaces and simplicity of design is commendable, it ignores the fact that the mouse and keyboard are themselves pretty cool. The speed of input on a keyboard leaves writing, text-to-speech and neat alternatives like Dasher in the dust. The mouse is an incredibly accurate pointing device – one that may be wasted on the majority of computing tasks, but indispensable for CAD and other realms where exactness is more important than fluidity.

The problem with the over-enthusiasm to ditch the old and embrace the new and super-simplified is that it rings hollow to anyone who knows where the old fits best. If you tell a typist that speech recognition is the way of the future, they’ll know that you’re wrong on that point – and consequently will be suspicious of claims in other realms they have less expertise in. Promising that NUI will overturn everything in computing, rather than just where our current solutions need work, threatens to turn people off of the whole thing.

That’s why voices like Bill Buxton’s are so important. In a video interview this year, he said he doesn’t even like the term NUI, let alone the suggestion that it will replace everything. Take the time to watch the video, it grounds the conversation about what comes next in something a little more useful, while still being optimistic about what can happen.  I like his term “Appropriate User Interface”, because 1. it’s less dramatic and 2. it provides a reminder that the application -as in where an interface is applied – is more important than the amazingness of the new- or recently-fangled technology we have available.

Aside from the hilarity at 10:50 when he freaks the presenter out by standing way too close, I was really intrigued by the idea of using distance-aware displays to dynamically scale things up or down depending on how far away you are. I knocked together a Flash-based experiment to that effect. Have a look at it below!

the closer your monitor is to your webcam the better, but it’s an interesting experience having the size of an image remain the same relative to your distance from the screen! It give the impression, like Johnny Chung Lee’s experimnts, of turning the screen into a window through which you just see another world, rather than just a picture. I’m very interested in how this will feel built ino a more robust framework.

Maybe not everyone.  Gamers seem to be in violent opposition to the idea of changing input methods.  On Slashdot, Gametrailers and other forums I still see people insist that the two-thumb controller is the best input method for platformers, a mouse the best method for Realtime Stragety (RTS), and a mouse and keyboard for First Person Shooter (FPS) games.

Gaming in 1990 (click to enlarge)

I find that reaction disappointing.  For a start, It’s probably not true.  Prior to Quake, most ‘gamers’ (though they did not exist in the same numbers or have the same recognition as a demographic) insisted that a mouse was an unnecessary and fiddly complication to the task of moving a character around, and that looking up or down could be done easily enough entirely with the keyboard. As strange as it seems, gamers and technophiles in general appear to be a very conservative lot.*

Second, even if it were true, and it is a possibility – it is only because the RTS, FPS and platformer genres have been forced to work as effectively as they can the available systems.  They’ve been designed with these input methods in mind, so of course they’re going to seem like the only option available.

Gaming in 2000 (click to enlarge)

When you change the range of input devices, though, the options broaden.  It’s not hard to imagine a shooter where the player can lean around corners, duck projectiles or operate machinery with a little more nuance than just a ‘use’ button.  With RTS games, we could issue two orders simultaneously with each hand, draw out formation plans with gestures or guide the attitude a subordinate AI officer commands with through a facial expression.

While I’m glad that “Motion Control” appears to be the bandwagon to jump on at present, I urge developers, gamers, and the public at large to see motion gaming as a ‘gateway drug’ to better and weirder things. We can turn this into a watershed moment in history not just for gaming but for computing in general, where we decide that the main challenges in computing today are with Human-Computer Interaction (HCI), rather than computation or processing.

Gaming in 2010 (click to enlarge)

One relatively recent breakthrough in Artificial Intelligence is the realization that agents are far, far more effective if they are properly embedded in the environment in which they operate.  That means having both consistency of – and plenty of – sensory information. Problems involving vision become (comparatively) simple when a number of camera views are added and combined as input, the problem of balance is made much easier if acceleration and force data is collected directly from multiple points rather than modelling ‘expected’ values internally.

It follows, then, that the task of making sense of what a user wants from a computer would be made easier by giving it more information.  What may be less obvious is that it works the other way around, too.  The more sensory input from a system we have, the more sense the system is going to make, and the more immersive that system becomes.  The “next big thing” in mobile devices after touch-screen interfaces is tactile feedback from interfaces.  The Nokia Haptikos display will try to simulate the press of a button by shaking on confirmation.

I welcome the shift in emphasis of the console vendors.  We need to ignore the grumbling from current gamers and game developers, who are very stuck in their ways.  These detractors are ignoring the trends in gaming that have led us to this point, and where these trends indicate that gaming will go.  I hope that by the time I have kids who are old enough to lecture, I can tell them to go outside and play a videogame.

Gaming in 2020 (click to enlarge)

* As an aside on the conservatism of technophiles, I remember that, at the ripe old age of 10, I was incensed at the suggestion that people were installing “sound cards” and other add-on hardware into IBM-PC compatible computers.  PCs were supposed to be generalized computing devices! If you wanted sound that badly, you should wait until the CPU could process it all in software! If I had heard that people were going to then go on and make cards that dealt exclusively with 3d graphics, I probably would have passed out from sheer indignation.  I was an unusual child.