Desktops in 3D

Advances in graphics technologies are pushing interface design in new directions

I
26

BY AARON WEISS

n 1993, moviegoers watched as dinosaurs stomped and chewed through the scenery of Jurassic Park. Their trail of destruction only began to subside when 10 year old Alexis scrambled into the parks control room and hopped onto a terminal, uttering the line, This is Unix, I know this! Viewers watched as precocious Alexis navigated a 3D GUI, flying through data objects on her way to ridding the world of genetically mutant T-Rexs. The dinosaurs and the Unix-savvy little girl may have been pure movie fiction, but in fact, the 3D interface was real.
Illustration by Hal Mayforth

DESKTOPS IN 3D

MARCH 2007

DESKTOPS IN 3D

DECEMBER 2005

27

Apples Expos uses the GPU to instantly scale all windows for quick selection.

But like so many 3D computer interfaces, SGIs fsn (pronounced fusion) file navigator for their Irix operating system was basically a technology demo. Its sole function was to represent a directory listing in three-dimensional space, coordinating physical proportions with file sizes. Not terribly usefulprecisely the albatross thats hung around the texture-shaded polygonal neck of the 3D GUI for years. Often too abstract and, despite their stated goals, too unintuitive, ambitious 3D desktop projects have tended to elicit the same response as Michael Jacksons 3D short film Captain Eo that played at Epcot Center: They look cool but whats the point? Will the 3D desktop interface ever mature? Actually, you may already be using one right now. Mac users with OS X certainly are. And soon, so will the rest of the

world, when they upgrade to Windows Vista, whether by choice or otherwise. 3D is already here, but it doesnt look quite like we thought it would. With the latest mainstream operating systems requiring ever more powerful graphics hardware and new cutting-edge projects pushing the 3D envelope even further, a question lingers: Is all this eye candy just a lot of empty calories, or a truly new way to interact with our PCs?

First Person Immersion

The graphical desktop is, of course, a metaphorit says so right in the name. The software desktop is organized like a physical desktop, except that it really isnt. The metaphor, only ever a conceptual launching point, has further blurred over time as the interface has become a thing unto itself detached from any physical experience. To programmers courageously wading

28

DESKTOPS IN 3D

MARCH 2007

into three-dimensional waters, the great temptation has been to redefine the user interface metaphor. Whereas a desktop recalls a flat, two-dimensional space, 3D programmers have fancied immersive environments. Prototype interfaces have served up metaphors from walking through a house to flying through abstract space. But immersive worlds tend to be confusing for basic computer operationseven if the tasks of manipulating files and applications do not cleanly map to the use of a physical desktop, they map even less cleanly to floating through an imaginary 360-degree dreamworld. Games, however, succeed brilliantly in an immersive world. Beginning with nowlegendary Castle Wolfenstein 3D, the first person shooter has become the primary visual grammar for video games. The Doom series, perhaps the most influential of early first person immersive games, even spawned a 3D desktop application called psDooma process monitor rendered inside the three-dimensional Doom world. But as usual, walking through a virtual world to annihilate monsters was a much more effective use of the technology than managing applications. While immersive 3D metaphors have failed to catch on outside the computer science lab, 3D gaming has exploded in popularity. While even games are not, of course, technically in 3Dour screens remain two-dimensional, after alleach new generation of games features visual effects that render depth with increasing realism. That realism is powered by hardware. Today, the market for graphics cards is pushed forward almost entirely by advances in 3D rendering technology. Each new generation of high-end graphics cards renders more pixels more quickly than the generation before. Todays low-end, sub$50 budget 3D cards can pump out over 1

billion pixels and over 100 million triangles per second. What does any of this have to do with 3D desktops? Everything.

Transparent Beauty
Although it has become commonplace to refer to todays powerful graphics cards as 3D cards, the name is misleading. To create the realistic effects craved by todays gamers, graphics cards have evolved into miniature computers unto themselves. Their specialized GPU (Graphics Processing Unit)the functional equivalent of a PCs CPUcan quickly blend images, apply lighting effects, and morph shapes. It is these skills modern desktops are turning to, offloading the processing burden which would otherwise weigh down the CPU and slow the machine to a crawl. When Apple released OS X in 2001, it became the first mainstream vendor to take a significant step forward in desktop architecture: The operating system relies on a GPU to render desktop graphics effects. Perhaps most striking among OS Xs visual trademarks is its use of transparency. Drop down menus are rendered over windows with translucency. The OS X dock is also transparent, letting windows appear behind it. By relying on the GPU of the graphics card included with Apple Macs, OS X can composite these transparency effects in real-time as the user works. But in going sheer, Apple also ignited debate. Few have argued that OS X is not aesthetically attractive, with transparency contributing to its appeal. But do transparent windows serve any function? When information appears in a window below another, it is not necessarily visually accessible. You can see the existence of text or pictures beneath the surface window, but it is often obscured by information in the top window.

DESKTOPS IN 3D

MARCH 2007

29

Task-switching in Windows Vista can be performed with a stacked set of windows with live preview.

Microsoft tore a translucent page from Apples book with its Aero Glass interface, a flagship feature of the new Windows Vista operating system. In Glass mode, which is supported only by graphics cards with sufficiently capable GPUs, the fat edges of desktop windows blur the contents of windows beneathas if looking through, wait for it glass. But the effect, while undoubtedly pretty, renders most of the information beneath the glass illegible.

A Matter of Scale
The GPU on a graphics card is a Swiss Army Knife of compositing functionality. Scaling images on the flyas in the OS X dock and its famous magnification effect certainly catches the eye. Sometimes called fisheye navigation by other vendors, the OS X dock instantly resizes icons as the mouse passes over them to indicate selectability.

Like transparency, dock magnification has quickly become a visual trademark of OS X, and has also exposed Apple to some criticism from interface design purists. Clearly, it is good for the user interface to provide feedback when an item is selectable. But some argue that with icon magnification, the visual effect is more style than substance. In fact, many Mac power users disable (or substantially reduce) the dock magnification effectsuggesting that it may be more stylish than practical. But we neednt conclude that real-time scaling cannot be productive. Apple proved this with Expos, introduced in OS X 10.4 to widespread acclaim. On a desktop cluttered with windows, one keystroke or hot corner instantly triggers Expos to resize and distribute the windows evenly, so that one may quickly be selected for focus. Not only does Expos resize all open windows in a flash, the scaled-down windows retain a live view of their content, such as a loading Web page. Windows Vista also leverages the PCs 3D graphics card to generate scaled, live

30

DESKTOPS IN 3D

MARCH 2007

previews. Vistas Flip 3D effect replaces the venerable alt-tab application switcher, arranging open windows into a threedimensional stacked set. Each window displays a preview of its contents. Even with their scaled-down previews, both Expos and Flip 3D are functional improvements over earlier window switchers. Today, a single application can spawn many windowsfor example, multiple browser windows or word processing documentsand simply knowing the name of an application attached to a window isnt enough. Some say that the next step for graphics-card-based scaling will be the resolution independent interface. Today, most screens feature a pixel density under 100 DPI. But as higher-density screens become available, todays operating systems will render awkwardly. You can increase the text size in a Web browser, for example, but this wont change the size of images on the page, or the menu items in the browser, or icons on the desktop, and so on. In the long run, operating systems will need to rely on vector rather than bitmap graphics to scale. The high processing cost of rendering vectors will shift even more work to the 3D graphics card to generate even basic display elements like widgets and text. Both Windows Vista and the upcoming OS X 10.5 Leopard take small steps toward vector-based displays, but both software and hardware remain a few generations from universally resolution-independent interfaces.

Through the Looking Glass

Glass seems to be a recurring metaphor with 3D desktop architecture, and Sun has joined the effort with their Java-based Project Looking Glass. Unlike Windows Vista and OS X, Looking Glass is platform-independent and runs in combina-

tion with the underlying native operating system. Looking Glass also goes further than either Vista or OS X in its vision of integrating 3D effects into the desktop experience. Applications running inside Looking Glass feature windows that are treated as four-sided objects. The edge or side of a window object features a title, like you might see on the spine of a book. Windows can be angled inward so that they can be arranged to take up less space while retaining visible content, a feature well also see in other forwardlooking 3D desktop projects. Perhaps the most innovative feature of Looking Glass is the notion that application windows have a back. You can flip, say, a Web browser window, like a pancake. Facing the rear side, the window content is reversed, which itself is not particularly useful other than for spatial consistency. More notably, Sun lets you attach notes to the window back, offering an intriguing new desktop functionality. By treating each window as an object in three dimensional space, Looking Glass actually offers new ways to interact with that object, something weve yet to see from the 3D effects employed by Apple and Microsoft. But Looking Glass is still more demonstration than production interface. For one, while its Java-based code will run on Windows, it does not support any native Windows applications. That means the only applications which benefit from Looking Glass features are those natively written for Looking Glass, of which there are very few. The Linux/Solaris version does integrate with native OS applications, but is still used primarily by those comfortable with experimental software. But because Looking Glass is completely open source, it may mature rapidly.

DESKTOPS IN 3D

MARCH 2007

31

Linux at the Edge

When it comes to extending the traditional desktop with GPU-greased enhancements ready for everyday use, OS X, Vista, and Looking Glass cannot yet compare to the Compiz/XGL or Beryl projects on Linux. Compiz/XGL and Beryl are two 3Denhanced window managers for Linux, both based on the same core code. Compiz was originally developed at Novell, and later forked into a secondary project now known as Beryl. In typical open source

but clearly superfluous, such as Beryls wobbly windowswindows which distort like elastic when moved around the screen, as if they are literally being dragged across a surface. Other effects, such as Beryls Opacity, serve a more functional purpose. With Opacity enabled, you can hover the mouse over any exposed part of a window and the whole window instantly becomes fully visible until you hover off. The best of Compiz and Beryls 3D effects, like Scale and Opacity, productively

ITS NO ACCIDENT THAT VISTAS 3 D AERO INTERFACE WILL PUSH HARDWARE SALESMICROSOFT KNOWS HOW TO KEEP PC VENDORS HAPPY.
fashion, these 3D desktops enjoy an embarrassment of riches, overflowing with eye candy. Perhaps the single most eye-catching feature of Compiz and Beryl is the desktop-ascube visual. Linux has long supported so-called virtual desktopsmultiple workspaces in which you can run applications. Compiz and Beryl render each desktop workspace as a side of a cube; thus, if youve set Linux to four virtual desktops, you can rotate a four-sided cube, each side containing an independent set of windows. While rotating, all windows on all sides remain live. Beryl includes its own version of Apples Expos, called Scale, which can instantly tile preview windows of all applications, either within a single virtual desktop or across them all. Both Compiz and Beryl include reams of optional eye candy effectssome are fun enhance managing many applications and windows. But because both projects are community developed, and eye candy is irresistible, both are subject to the criticism that they are ninety-percent sizzle and tenpercent steak. Where Compiz and Beryl are the flashier Linux 3D desktops, their more sensible cousin could be Metisse. Developed in France by the In Situ Project, Metisse is a windowing framework with a focus on human computer interaction over pure eye candy. Available now on a free live CD distributed with Mandriva Linux 2007, Metisse features a variety of ways to interact with windows as if they were foldable objects. Windows can be tilted inward, as in Project Looking Glass, but also folded partially over, like pieces of paper. Doing so offers new ways to arrange and manage windows while keeping pertinent information available and accessible.

32

DESKTOPS IN 3D

MARCH 2007

The Literal Desktop

Most of the 3D effects being used by Apple, Microsoft, and Linux are far removed from the original desktop metaphor. After all, paper documents arent particularly transparent. Whereas experimental 3D interfaces have run aground trying to build immersive new metaphors, Anand Agarawalas BumpTop takes the 3D desktop in the most literal direction yetrecreating a physical desktop experience. The BumpTop interface is built on a plane, like the top of a desk, and represents objects in piles just as you would see on a physical deskpiles of memos, piles of mail, piles of pens, and so on. The central mechanism behind the BumpTop is that it is physics enabled. So, like in the real world, if you drag an object into another, they will collide, and one moves the other out of the way. It is by manipulating objects this way that you accomplish tasks on the BumpTop. Objects are represented on the BumpTop by how they look. So, for example, you can quickly see all PDF documents, and toss them into a pile. Unlike the physical world, in the BumpTop world you possess enviable, almost magical powers. You can lasso together a group of objects and manipulate them in a bunch, as if waving a magic wand. BumpTop, although a university thesis project, impressively extends the desktop metaphor into a hyper-literal, rather than increasingly abstract, three dimensions. In doing so, though, it raises the question as to whether replicating the physical world on the computer really adds to productivity. Maybe, as the BumpTop seems to illustrate, we simply inherit many of the limitations of the physical world.

top will arrive because its already here. When Apple designed OS X to utilize the graphics card GPU to render common tasks, the barn doors flung open and realtime compositing effects entered the mainstream. But without powerful GPUs at commodity prices, Apple would have had no means to that end. Credit the PC, and in particular its gaming vendors, with creating a market for 3D graphics cards to continually push the technology envelope. User interface designers wonder how many 3D desktop effects contribute to real productivity. The question may as well be metaphysical, though. We so often hear about improving the user experience, but is the physical attractiveness of a platform not part of its experience? Regardless, the 3D desktop will continue to evolve because its good for business. Ask many veteran video game enthusiasts and theyll note, sometimes lamentably, that most of todays games essentially rehash the same core challenges seen in games twenty years ago. But every year the games look better and sell more. They sell more games and they sell more hardware. Windows Vista has taken criticism for requiring newer hardware to support its flashy Aero interface. But of course, that Vista will push hardware sales is no accidentMicrosoft knows how to win the good graces of the PC vendors on whom it relies. The jury may be out as to whether 3D desktops are more productive to use, but Apple and Microsoft are wagering that 3D is a very productive sell.

PERMISSION

TO MAKE DIGITAL OR HARD

COPIES OF ALL OR PART OF THIS WORK FOR PERSONAL OR CLASSROOM USE IS GRANTED WITHOUT FEE PROVIDED THAT COPIES ARE NOT MADE OR DISTRIBUTED FOR PROFIT OR COMMERCIAL ADVANTAGE AND THAT COPIES BEAR THIS NOTICE AND THE FULL CITATION ON THE FIRST PAGE.

TO

COPY

OTHERWISE, TO REPUBLISH, TO POST ON SERVERS OR TO REDISTRIBUTE TO LISTS,

Form and Function

Theres no question whether the 3D desk-

Aaron Weiss is a technology writer and Web developer shivering in upstate New York, as well as human proprieter of livenudecats.com

REQUIRES PRIOR SPECIFIC PERMISSION AND/OR A FEE.

ACM 1091-556/07/0300 $5.00

DESKTOPS IN 3D

MARCH 2007

33