See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220876899

Foldable interactive displays

Conference Paper · January 2008

DOI: 10.1145/1449715.1449763 · Source: DBLP

CITATIONS READS
67 1,342

3 authors:

Johnny C. Lee Scott E. Hudson

Carnegie Mellon University Carnegie Mellon University
16 PUBLICATIONS   1,186 CITATIONS    252 PUBLICATIONS   8,833 CITATIONS   

SEE PROFILE SEE PROFILE

Edward Tse
SMART Technologies
42 PUBLICATIONS   720 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Edward Tse on 29 July 2014.

The user has requested enhancement of the downloaded file.

 Foldable Interactive Displays
Johnny Chung Lee, Scott E. Hudson Edward Tse
Human-Computer Interaction Institute Smart Technologies
Carnegie Mellon University 1207-11 Avenue SW, Suite 300
5000 Forbes Ave., Pittsburgh, PA 15213 Calgary, Alberta, Canada T3C 0M5
{johnny, scott.hudson}@cs.cmu.edu edwardtse@smarttech.com
ABSTRACT
Modern computer displays tend to be in fixed size, rigid,
and rectilinear rendering them insensitive to the visual area
demands of an application or the desires of the user.
Foldable displays offer the ability to reshape and resize the
interactive surface at our convenience and even permit us to
carry a very large display surface in a small volume. In this
paper, we implement four interactive foldable display
designs using image projection with low-cost tracking and
explore display behaviors using orientation sensitivity.
Author Keywords
Foldable displays, interactive, mobile, projection,
augmented reality, orientation sensitivity, privacy.
ACM Classification Keywords
H5.2 [Information interfaces and presentation]: User
Interfaces. H5.1 [Multimedia Information Systems]:
Augmented Reality. Figure 1 – Foldable fan display with stylus input.

INTRODUCTION this paper, we explore this concept of inactive foldable

In the realm of science fiction, future display technology
often depicted as holographic surfaces that float in thin air.
Sometimes these displays can be summoned at will in
proximity to a person’s body, can be changed in size and
shape to fit the desired usage, can be collapsed or dismissed
in an instant if the user needs to tend to some other activity,
and of course, can support interactive input. While modern
displays have become thinner, higher in resolution, and
provide input using a stylus or touch sensitivity, we still are
a long way from achieving these technological visions.
Many of the displays we see in hand-held devices today are
small LCD displays of fixed shape and size. In this respect,
they are insensitive to the desires of a user or the needs of
an application. Ideally, we would like displays that we can
dynamically reshape or resize to suit our desired usage,
similar to the way we might handle a newspaper, or simply
so that we are able to fit a large display into our pocket. In
Permission to make digital or hard copies of all or part of this work
for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage
and that copies bear this notice and the full citation on the first page.
To copy otherwise, or republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee.
UIST’08, October 19–22, 2008, Monterey, California, USA.
Copyright 2008 ACM 978-1-59593-975-3/08/10...$5.00.
displays and create a number of working prototypes such as
the one shown in Figure 1.
Emerging technologies such as electronic paper and organic
light emitting diode (OLED) displays are expected to
provide some degree of flexibility. However, current
prototypes remain quite rigid and are typically rectilinear.
This prevents them from becoming truly foldable in the
sense that we think of paper as being foldable. Additionally,
performing input on such flexible displays is an entirely
separate technological hurdle. The approach we use in
exploring flexible displays is by augmenting the appearance
of passives surfaces with image projection [1, 2]. This
allows us to combine the flexibility and minimal weight of
plain paper or fabric with the dynamic content capabilities
of a computer display creating a coherent and fully
functional user experience. PaperWindows utilized a
similar approach to explore interaction techniques with
sheets of paper as if they were digital displays [4], but
focused mostly on flat paper interactions in a tabletop
scenario and relied on a high-cost motion tracking system
for location discovery. Our work demonstrates foldable and
re-shapeable semi-rigid hand-held displays using an
affordable, low-cost location tracking technology.

287
 Figure 2 – Foldable display shapes (left to right): newspaper, scroll, fan, and umbrella.
Red dots indicate the location of the LEDs use for tracking in these prototypes.
Tracking
Tracking is accomplished using infrared (IR) LEDs
embedded into the foldable surface and the infrared dot
tracking capabilities of the PixArt camera within the
Nintendo Wii remote. This camera contains hardware blob
tracking up to four points simultaneously with a resolution
of 1024 by 768 at 100Hz. Since the computer vision
tracking is performed by hardware in the camera, this is a
low-cost, easy to implement solution that provides high-
resolution, low-latency tracking. When positioned adjacent
to a projector, the camera tracking data can be calibrated to
the image location resulting in spatially augmented reality
on moving surfaces [5]. However, camera based tracking
has limitations such as the number of distinct points that
can be reliably tracked simultaneously, the inability to
provide point identity, and manual calibration with the
projected image.
The four point limit of the PixArt chip reduces the
complexity of our foldable geometries requiring simplifying
assumptions, such as requiring shapes to have fairly high
folding symmetry or equally sized square panels. We take
advantage of the semi-rigid properties of our designs and
place the IR emitters at strategic locations such that we can
fit a geometry model to the data. This model provides
estimations of the display edges and corners which may not
be explicitly tracked. Assumptions are made regarding
display starting orientation, point ID, and the range of
movement allowing the model to fit the surface using a
small number of points. The fewer points that are available,
the stronger these estimations and assumptions must be.
However, as the technology evolves providing larger
numbers of trackable points, many of these assumptions can
be eliminated resulting in more robust performance. If the
projector was capable of presenting both visible application
Figure 3 – Foldable display prototypes at various stages of
content and non-visible structured light patterns [6], large
expansion (top to bottom): newspaper, scroll, fan, & umbrella
numbers of points could be tracked simultaneously without
ambiguous identity further reducing the number of FOLDABLE SHAPES
assumptions and spatial limitations. In this section, we present four foldable display designs.
The LEDs in the display surface run for several hours using This is, of course, not an exhaustive list. However, we
a small rechargeable battery pack. Since infrared LEDs believe they present a number of expansion and collapsing
emit non-visible light, the LEDs appear as small black dots behaviors likely to be used in a typical foldable display. For
5mm in diameter. The LEDs can also be placed beneath a each description, please refer to Figure 2 for an illustration
translucent surface to hide their visual presence entirely. As and Figure 3 for images of the working prototype.
with any optical based tracking system, occlusions are a Newspaper
problem but could be address using additional cameras or One of the most common formats in which we interact with
location estimation using remaining visible points. large sheets of printed material is a typical newspaper.

288
 Figure 4 – Orientation sensitivity behaviors (left to right): double-sided display surfaces can react differently depending on the
direction they are flipped, simulated lenticular can change the document view depending on the angle of viewing in a hand-held
display, or a tabletop scenario where tilt angle may correspond to different privacy states: private, public, excluded.
Sometimes referred to as the broadsheet format, these large
sheets of paper are folded in half vertically and then again
horizontally allowing a variable visual area ratio of 4 to 1.
Additional folds can be added to further increase the
magnitude of variability in visual area. In our prototype, we
use two folds to support viewing half a page up to two full
pages side by side. The user can gracefully increase or
decrease the viewing area simply by unfolding or folding
the display. Tracking, when viewing a half-page, uses only
two IR emitters defining the top edge of a rigid square
panel. The first fold is defined by a third point in the bottom
left corner becoming visible below. The second fold is
defined by a fourth point appearing in the top right corner.
Scroll
While less common today, large printed material was once
transported and viewed in the format of scrolls that could be
unrolled. This allowed individuals to not only customize the
amount of visible area, but also the location of that area
within a long document – hence the concept of “scrolling” a
window in a typical GUI environment. By creating a digital
display scroll we can change the size and aspect ratio of the
viewable area quickly and easily exposing more of the
application content. This design can also be collapsed into a
relatively small form factor for storage. This is tracked
using an IR emitter in each corner and the display is
assumed to begin in an upright orientation.
Fan
Folding fans are perhaps one of the best examples of a
device that must be very large in surface area to be
effective, usable by one hand so that the other hand is free
to perform a task, and collapsible for easy storage in a
pocket. Coincidentally, these properties are also desirable in
a mobile display technology. As a result, this design may be
one of the most practical for foldable displays in a mobile
scenario. In our prototype the ratio of display area from a
fully expanded to fully collapsed configuration is
approximately 20 to 1 ranging from 100 square inches to a
small strip. The elongated strip can be used to display status
messages, rolling text, or progress bars similar to a portable
music player. Some folding fans designs allow full 360
degree expansion creating a circular display area. The fan
format can either be used in full or partial expansion to vary
the amount of screen area desired. While we used a pleated
folding fan for our prototype, folding fans can also be
composed of parallel slats resulting in a nearly planar
289
display surface minimizing distortion. Two IR emitters
define the outer and inner radius of the display area. The
hinge location is estimated using the known physical
proportions of the fan. A third IR emitter identified the
sweep of the display along an ellipsoid.
Umbrella
Another common example of expanding and collapsing a
large surface is a parasol or umbrella. These surfaces can
frequently be operated by one hand using spring loaded
designs and can produce a very large surface area very
quickly. Depending on the culture of origin and intended
purpose, umbrella and parasol designs vary from parabolic
bell shapes, to conical, to nearly planar. Distortions due to
non-planar surfaces can be compensated for if the geometry
is known before hand. An umbrella design may perhaps not
be the most ideally suited shape for interactivity due to the
central perpendicular column of the handle. However, the
surface area change ratio is very dramatic making it
potentially attractive for certain applications and lends itself
to rotational input. The handle also provides an optically
plausible location for a projection and tracking device for
true mobility. When all four IR emitters are visible, a plane
is defined which orients a parabolic model.
ORIENTATION SENSITIVITY
Though we track gross movements for the purposes of
projection and folding, we can also respond to flipping or
subtle display movements to create orientation sensitive
behavior. The following behaviors are illustrated in Figure
4 and images of the working prototype are shown in Figure
5.
By monitoring the persistent visibility of edges, we can
detect when the display has been flipped and in which
direction. This is accomplished by tracking emitter
visibility and motion modeling to determine probabilistic
movement. Emitters are placed along the edge such that
they remain visible when viewed from the side but back
facing edges will result in disappearance. This provides a
way to create lightweight double-sided display surfaces
simply by projecting different content on each side.
Flipping the surface in different directions (left, right, up,
down, or diagonally) can trigger events for document
navigation or view manipulation [3].
We can react to more subtle tilting movement of the surface
altering the view depending on the angle at which is it held.
 This is done by monitoring the ratio of the edges of a sem-
rigid surface. We refer to this as a simulated lenticular
screen and can respond to multiple axes of orientation
change.
In a multi-user scenario, horizontally tilting or turning the
display to another person (assuming the computer is aware
of viewer locations) can trigger different document views.
If placed on a table top, vertical tilting may correspond to
the implicit privacy states of the display depending on its
expected visibility. Tilting the display toward yourself
might indicate a private state, placing it flat would indicate
a public state, and tilt it away might indicate an excluded
state, possibly useful for presentation or gaming scenarios.
Use of this capability will depend on the application
scenario of the table top system.
While it would be possible to have the system react to
additional degrees of rotational freedom as well as three
degrees of translational movement, having such complex
display behavior would be highly application dependent.
For example, six degree-of-freedom display tracking would
be appropriate for creating a view portal into a virtual 3D
environment but unnecessary for many 2D GUIs.
INTERACTIVITY Figure 5 – Orientation sensitivity prototypes: (top) double-
By tracking additional dots over those embedded into the sided flip direction, (middle) vertical and horizontal simulated
display surface, we can track a stylus for input shown in lenticular in a hand-held surface, and (bottom) a tabletop
scenario where tilt correlates to different privacy states.
Figure 1. Since our fan design used only three IR emitters,
we were able to use the fourth point to represent an input
manner. As tracking technologies improve, surface
stylus. The stylus contains a button to activate the emitter
modeling assumptions can be reduced improving
providing a passive method of detecting clicking and
robustness. With the emerging flexible display and portable
dragging. The location of the cursor is mapped to the 2D
projection technologies, foldable designs may become
location on the foldable display. Alternative style designs
practical in portable consumer electronics in the near future.
could utilize a tip switch, but this will vary performance
when using different display materials. This technique ACKNOWLEDGMENTS
provides an easy way to obtain interactivity on all of the This work was supported in part by grants from General
surfaces described including their double-sided variants. Motors, the Intel Research Council, and the National
High-speed IR data transmission could be used to transmit Science Foundation under Grant IIS-0713509
additional data from the stylus such as left and right click, REFERENCES
or orientation data. While it would be possible to support 1. Badyopadhyay, D., Raskar, R., Fuchs, H., “Dynamic Shader Lamps:
multiple cursors, a camera based approach may lead to Painting on Real Objects.” IEEE ACM Int. Symposium on Augmented
tracking confusion when the pens approach each other or Reality (ISAR), New York, 2001.
the emitters defining display boundaries. However, 2. Brown, L. et al, "A Widget Framework for Augmented Interaction in
SCAPE" Proceedings of the ACM Symposium on User Interface
projector-based tracking would allow many cursors to be Software and Technology, October 2003.
used simultaneously without ambiguity [6]. 3. Chen, N., Guimbretiere, F., Dixon, M., Lewis, C., Agrawala, M.,
"Navigation Techniques for Dual-Display E-Book Readers" CHI
CONCLUSION 2008.
In this paper, we explored the concept of interactive 4. Holman, D., Vertegall, R., Altosaar, M., Troje, N., Johns, D., "Paper
foldable displays by presenting four functional prototype windows: interaction techniques for digital paper" Proceedings of the
designs featuring different folding behaviors. The displays SIGCHI conference on the human Factors in computing systems,
2005.
react to the user’s movement, expansion state, orientation, 5. Gupta, S. Jaynes, C., "Active Pursuit Tracking in a Projector-Camera
and input styli. Foldable displays offer the ability store System with Application to Augmented Reality" IEEE Computer
large display surfaces in a small volume allowing users to Society Conference on Computer Vision and Pattern Recognition,
quickly expand and collapse them when convenient. 2005
6. Lee, J., Hudson, S., Dietz, P. "Hybrid Infrared and Visible Light
While the current state of projection technology is not yet Projection for Location Tracking" Proceedings of the ACM
Symposium on User Interface Software and Technology, October,
practical for supporting spatially augmented reality in truly 2007.
mobile scenarios, this exercise allows us to explore
interacting with high-fidelity prototypes in a low-cost

290

View publication stats