Minority Report-like interfaces from TED

As readers of my blog may know, I'm a fan of multi-touch and similar gestural interfaces. Here's a recent video from the TED conference, from a consultant for the movie Minority Report, demonstrating interacting with data via gestures:


As you may have noticed, I'm fascinated with multi-touch interfaces. I just saw an interesting concept that proposes a desktop-based multi-touch system that avoids the need for touching a screen directly (which has issues of fingerprints, muscle strain, and obscuring the view), plus an interesting desktop usage metaphor.

I'm not entirely sold on it, but it definitely has some potential. I'm really not sure how well the proposed windowing system would scale — I currently have 23 windows open between all my apps, and often have more; their system of scrolling through them could be cumbersome, though the zooming out overview might be okay.

They also seem to reserve most multi-touch gestures for managing the windows, which could limit what apps can do with them.

Anyway, check it out; it's a fascinating concept, and a good description of some potential issues with more conventional approaches:

10/GUI from C. Miller on Vimeo.

WWDC 08 predictions

Unfortunately I'm not attending WWDC this year. I'm not currently working on or immediately planning an iPhone app, and don't expect anything much new for Mac OS X, so it wasn't worthwhile to go this year. Maybe next year?

I'm sure I will write some iPhone apps in the future, though; it's an exciting platform, and I'm really looking forward to getting a new iPhone once they're released.

But speaking of releases, time for some prognostication.

The leading expectation for WWDC 08, of course, is the 3G iPhone with version 2.0 software and the software development kit (SDK). I'm confident that this will eventuate. I'm not sure whether or not the new iPhone model will be immediately available, or just announced for pre-order and delivery later in the month (or even later). If I had to guess, I'd say it'd be released no later than the end of June... but I'm hoping for immediate availability.

There are also questions of whether or not the updated iPhone will be thinner or thicker than the current model, what memory size it'll have, if there will be multiple models (perhaps a cheaper 2.5G and more expensive 3G model), coloring, form-factor, etc. I would guess thinner, double the memory, and only a 3G model (with a preference to switch between 2.5G and 3G).

Another rumor that has been popular recently is an unusual update to Mac OS X to version 10.6, code-named "Snow Leopard". This would be unusual in that it is supposed to not include any major new features, but just concentrate on tidying up the code base, improving performance and stability... stuff that is normally the realm of bug-fix releases.

But it is supposed to also drop PowerPC support. This would make a certain amount of sense — it'd allow throwing away lots of code, and simplifying many things. But it might be a little too soon for such a drastic change; there are still plenty of perfectly good PowerPC machines out there (I have a few in active use).

There have been rumors that 10.6 would change Carbon support in some way. Some thought it would drop Carbon entirely, or Carbon UI, but I don't think that is realistic. There are still many Carbon apps out there, including big ones from Adobe, Microsoft, and others. What I could see happening, though, is (as Gruber says) adding Objective-C wrappers around framework calls that are only available via Carbon currently. That would certainly be very welcome; as a Cocoa programmer, it can be mildly distasteful to have to drop down to Carbon to implement some functionality, though it's certainly not the end of the world.

Another popular rumor is that .Mac will be overhauled and renamed, perhaps as "Mobile Me". This has been fueled by people noticing that is owned by Apple, and seeing the text "Mobile Me" referenced in resources. This does seem pretty conclusive, though I can't say I particularly like the name. On the other hand, .Mac has always been a silly name, so Mobile Me isn't any worse. It does certainly make sense to rebrand it to avoid reference to Macs, now that Apple has a major non-Mac platform in the iPhone.

Finally, some people are predicting a new multi-touch device, perhaps some sort of tablet or Newton-like form factor. I'd certainly welcome that, but am rather skeptical that such a device would be introduced now. I'm sure Apple has a few such devices in development, even if only as experimental projects, but introducing one now would distract from the new iPhone, unless it were positioned as a "super-iPod touch" kind of device, running the mobile OS X. I would really like to see a multi-touch Mac tablet... but that seems even less likely at this stage.

I would be very surprised if any Mac hardware were announced. WWDC isn't traditionally the venue for hardware releases; last year Apple released updated MacBook Pros a week before WWDC, rather than waiting a week. At its core, WWDC is for developers, talking about the OS. That's the way it should be.

It'll certainly be very interesting to see what is announced. Maybe Steve will surprise us all. Let's all play WWDC bingo! The real surprise would be if Steve doesn't say "boom". :)

Stevenote thoughts

Well, that was an interesting Stevenote. Unfortunately, I wasn't able to be there this year. I enjoyed Macworld last year, but couldn't make it again. So I had to settle for reading the live-blogging coverage via They did a good job, though.

As expected, it didn't top last year's... it'd be hard to top the iPhone introduction! The rumor sites pretty much nailed the announcements, too.

The biggie of course was the new MacBook Air, Apple's re-entry into the sub-notebook market. I say "re-entry" as I would count their previous PowerMac Duo models as their first attempt at this market. Those were popular machines in the day, but the new Air is certainly a vast improvement. I personally am happy with my 17" MacBook Pro, and need the Pro power and large screen, since I use it as my primary development machine. But I'm sure the Air will be a popular model with people who want a compact lightweight laptop.

Perhaps the most interesting feature of it for me is the multi-touch trackpad. Readers of my blog will know that multi-touch is an interest of mine, so I'm pleased to see it come to the MacBook line. I only hope that they make the new pinch, rotate, etc gestures available for other MacBooks in the next OS update.

Another new product is Time Capsule, an AirPort Extreme base station with a built-in hard drive for seamless network-attached storage. This promises to be a boon for people with laptops, to avoid the hassle of having to plug in and dismount an external drive to do Time Machine backups. Of course, a concern is what happens if the built-in drive dies, as hard drives are wont to do, but presumably it wouldn't be too hard to swap it out, and it does include a USB port that supports an external drive.

The Apple TV enhancements, including movie rentals, seem worthwhile improvements. I have a Mac mini hooked to my TV, so don't really have a need for an Apple TV, but it seems a good device for many others. I can see lots of potential with this little gadget. The rentals make a lot of sense. Most movies are only worth watching once, so it's silly to buy them. We subscribe to NetFlix, a convenient DVD-by-mail service, but instant download of HD movies would be much more convenient.

Lastly, the new iPhone update seems to include many welcome improvements. The new location feature goes a long way towards GPS functionality, while the ability to save web clips to the home screen, and rearrange the home screen, both seem excellent additions. Shockingly, I don't have an iPhone yet, as I'm stuck into an existing contract till June, but you can be sure that my wife and I will be getting them once we're free of that... hopefully once the 3G rev is out, too. :)

All in all, nothing too surprising this time, but not a disappointment, either.

Multi-touch on the desktop

Craig Hockenberry wrote thought-provokingly about multi-touch interfaces on the desktop (via Gus Mueller).

This is a favorite topic of mine; as I've written before, I look forward to the day when multi-touch comes to desktop (or portable) computers.

One of his objections was the vertical orientation of traditional displays:

If you’re one of the people who think that a multi-touch monitor is a good idea, try this little experiment: touch the top and bottom of your display repeatedly for five minutes. Unless you’re able to beat the governor of California in an arm wrestling match, you’ll give up well before that time limit. Now can you imagine using an interface like this for an eight hour work day?

But he quickly counters that objection with what I feel is the obvious answer: a touch-based interface needs to be at a comfortable angle. I envision a desktop multi-touch surface at a 30-degree angle, or less, from the desktop: as he says, like a classic drafting table. Perhaps there won't be a distinction between desktop and notebook computers anymore, or perhaps the computer will be in two parts: a tablet-like mobile portion, which docks into and rests on a wedge-like stand on your desk, which adds additional functionality (kinda like the old PowerBook Duo and DuoDock).

The multi-touch screen would be the entire interface (other than perhaps some auxiliary buttons like brightness, volume, etc). It would obviously replace the mouse/trackpad, but would also replace the keyboard, using an onscreen keyboard instead. Yes, tactile feedback is an issue, but as many people have reported with their iPhones, it's possible to get used to typing without it; and there are ways to provide feedback, like the iPhone's magnified view of pressed keys, sounds, vibrations, and other ideas being worked on.

Hockenberry also raises a valid point regarding the precision of a mouse pointer vs a finger:

But even if there was a solution to the ergonomic issues, there would be problems mixing mouse-based applications (with small hit areas) with touch-based inputs (and large hit areas). Touch-based UI is not something you just bolt onto existing applications—it’s something that has to be designed in from the start.

Certainly an important consideration. However I would argue that most applications could be modified to support larger hit areas in sensible ways without too much difficulty - though in some cases major redesigns would be needed. Just have a look around the controls in your favorite apps, and think about how easy it would be to "click" on one with a finger, without activating a nearby control. In most cases, controls are spaced out enough for it to not be a problem, but some, like Photoshop, would require either optional support for a stylus (which Apple probably wouldn't be in favor of), or a finer on-screen control (perhaps like the iPhone's magnifying glass). I'm sure apps designed from the ground up with multi-touch in mind would be better... but migration is certainly possible. And yes, resolution independence should help. If you've got big fingers, you just scale everything up to a comfortable level.

I really believe that multi-touch is the way of the future, and will be coming for Macs in due course. But Apple being Apple, they will do it right, with as smooth a migration path for developers and users as possible.


So, I'm at WWDC currently. I've enjoyed meeting fellow developers (and several Dejal product users) around the conference center, at the sfMacIndie event, and the WWDC reception tonight.

I managed to get an okay seat for the Stevenote, a little back from the center of the room... but in a good position to see the repeater screens. I thought it was an interesting keynote, but with few surprises. I know some people are disappointed, but perhaps they had too high expectations?

So, how'd I do with my predictions? Let's see:

  • No hardware announcements: I was right with this one.
  • Possibly updated displays: nope... but not too surprising.
  • We might see a demo of iLife 07 and/or iWork 07: nope.
  • Steve will no doubt mention the iPhone, but there won't be a SDK: yep, called that one. I know this is disappointing for many people, but as I said, that may come later.
  • Steve will spend most of his time talking about Leopard: sure did.
  • The Finder might be rewritten: yes, though it seems more of a cosmetic overhaul than a full rewrite.
  • I've love to see more multi-touch features in the OS, but I don't think we're ready for that yet: yeah, no mention of that... oh well, it was a faint hope!
  • Maybe improved syncing features, to fit in with the iPhone: I seem to recall some mention of this?
  • The ZFS thing has been a popular rumor of late, but I have no idea whether that will eventuate: no mention of this, so perhaps it didn't... or perhaps Steve was annoyed by the "leak".
  • Probably some cosmetic changes in Leopard - using the newly fashionable dark metal styling for all windows, and other UI improvements: yes indeedy. I do like the dark metal, so a unified OS-provided look is a very good thing.
  • Probably no major new programmer APIs: so it would seem.

I seem to have done alright. Of course, my predictions were based on an aggregation of rumors, so hardly a reflection of my own prognostication abilities.

Anyway, on with the WWDC week... though of course I can't write about anything else discussed in the sessions.

A great multi-touch video

As I've previously discussed, I really believe that multi-touch is the way of the future. Here's another great video like Jeff Han's previous one, but on a bigger scale:

(Via MacRumors.)

iPhone: a hint of Macs of the future

Last night I was imagining what the Mac of the future might be like... and I think the iPhone gives us some idea... and perhaps more so Jeff Han's multi-touch demo at TED.

I've always thought that a tablet Mac would only have limited appeal - great for real estate agents, medical professionals, and some others, but impractical for everyday use by most people. But Jeff Han's demo and the iPhone have me rethinking that.

The main issue, of course, is input: a finger or stylus is fine as a pointing device, much like a trackpad or mouse... but text input isn't as practical. Sure, Apple has Inkwell, which supports handwriting recognition, but typing on a keyboard will probably always be faster for most people.

But while there is definitely some advantage to the tactility of a hardware keyboard, that may be mitigated by the versatility of a software keyboard - displayed on-screen.

I think that this may well be the direction Apple will head.

I imagine a future iMac as a 30" panel angled at about 45° from horizontal (adjustable), with the computer guts hidden underneath. There is no keyboard, no mouse - just a large screen right in front of you, like an architects drafting table. You interact with it with just your hands - no stylus or other hardware.

Like in the iPhone, you can scroll with the flick of a finger, "click" or double-click just by touching, and use multi-touch gestures to zoom, move, resize, and even rotate the screen content.

As in the picture manipulation in Han's multi-touch demo, windows in the Mac OS X of the future would float in three dimensions. You can zoom windows forward or back, drag them around (perhaps via touches of their titlebar or empty space, like modern textured windows), etc. The windows scale smoothly via resolution independence. There could be a button somewhere on the screen (or a hardware "home" button like on the iPhone) to show all of the windows, like exposé, allowing quickly finding a specific one.

Controls within windows would work the same way: flick scrolling, pinch zooming, finger dragging, and more.

As I mentioned, the biggest issue for me has always been the keyboard... but I'm coming around to the view that a software keyboard could be an entirely feasible replacement. The keyboard could zoom into view when you need it, vanish when you don't, and be reconfigured to suit the application. A numeric keypad with special function keys in calculator and spreadsheet apps, a full qwerty keyboard in a word processor, and other variations. They keyboard could even be scaled and moved around as needed. To compensate for the lack of tactile feedback, it could play tapping sounds when keys are pressed, or even speak the keys or words typed.

It'd probably still need to have menus at the top of the screen, but maybe some sort of contextual replacement could be devised. Similarly, it might still have the Dock, but it'd be zoomable and much more flexible.

Reminds me of Apple's Knowledge Navigator concept video... from 20 years ago.

We could even carry it further, perhaps for more portable Macs: perhaps something like a small unit that projects the keyboard and the user interface, using spacial sensing to detect your fingers. This would allow a pocket-sized device to have not only a virtual keyboard, but a virtual screen as well, perhaps several times larger than the device itself.

Of course, none of this is new... and the technology all exists today. If anyone can put it all together in a way that works, Apple can.

I can't wait.

Syndicate content