Works-in-Progress CHI PLAY'19, October 22–25, 2019, Barcelona, Spain

HyperCubes: A Playful Introduction to

Computational Thinking in Augmented
Reality
Anna Fuste Chris Schmandt Abstract
MIT Media lab MIT Media lab We present HyperCubes, an Augmented Reality (AR) plat-
75 Amherst St. 75 Amherst St. form to foster computational literacy. Using paper cubes as
Cambridge, MA 02142, USA Cambridge, MA 02142, USA
AR markers and spatial tracking, the user becomes familiar
afuste@media.mit.edu geek@media.mit.edu
with sequences of instructions as coding blocks. We lever-
age spatial cognition as a means to improve understanding
of procedural and sequential models. We have performed
two pilot studies for an iterative and user centered design
of the platform. With a final qualitative user study we ad-
dress engagement levels and the educational potential of
the platform. We argue that by using spatial cognition and
the flexibility of the AR medium, a playful introduction to ba-
sic computational thinking concepts can be presented in
late elementary school and middle school children.

Author Keywords
Augmented Reality; Computational Thinking;K-12 Educa-
tion; DIY; Paper Crafts; Programming; Spatial Cognition.

CCS Concepts
Permission to make digital or hard copies of part or all of this work for personal or •Human-centered computing → Mixed / augmented re-
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
ality; •Applied computing → Interactive learning envi-
on the first page. Copyrights for third-party components of this work must be honored. ronments; •Social and professional topics → Computa-
For all other uses, contact the owner/author(s).
CHI PLAY EA’19, October 22–25, 2019, Barcelona, Spain.
tional science and engineering education; K-12 education;
Copyright is held by the author/owner(s).
ACM ISBN 978-1-4503-6871-1/19/10.
http://dx.doi.org/10.1145/3341215.3356264

379
Works-in-Progress
Introduction
Computational thinking is a basic skill that should be in-
cluded in the educational curriculum in schools [19]. It is
also a learning framework that can incorporate other basic
subjects such as mathematics or science [35]. The major-
ity of the tools that are being developed for computational
thinking are screen-based online tools that often make no
use of space and embodiment, which are regarded as crit-
ical to the learning process by well-known cognitive and
Figure 1: Student playing with the
psychological theories [2] [21] [34].
HyperCubes application
We present HyperCubes, an AR platform for children that
offers a playful introduction to basic computational thinking
skills, by spatially orienting handmade paper cubes. Hy-
percubes is the first AR computational literacy platform that
allows for a room-scale visuo-spatial experience, and hence
a space large enough for group learning and collaboration
using spatial cognition. With tangible blocks and spatial
tracking, the user can freely move around and create con-
figurations for a flow of virtual objects in the environment.
Related work
Numerous platforms have been developed to teach compu-
tational literacy [26]. Here, we review the different platforms
and their contributions to the educational curriculum.
Digital Platforms
One of the initial platforms for learning computational think-
ing was Logo [23]. Children drew on the screen guiding a
digital pointer, in the form of a turtle, through code. Since
then, other platforms with similar approaches have been
developed. Agentsheets [27], Etoys [16] or Alice [6] are ex-
amples of digital platforms that do not require traditional
programming skills in order to foster computational literacy.
Scratch [28] is a block-based visual programming language
and online community targeted primarily at children. Other
CHI PLAY'19, October 22–25, 2019, Barcelona, Spain
block-based visual programming tools for mobile computing
have been developed such as the MIT App Inventor [24].
Tangible Platforms
Past research has demonstrated, however, how a tangible
interface is significantly better than a graphical user inter-
face when teaching concepts to children [33] [13]. Algo-
Block [32] is a tangible platform to teach coding skills to
children using physical blocks. In Tangible Programming
Bricks [20] the user stacks electronic blocks in order to con-
trol toy cars. In Quetzal and Tern [12] the user arranges
physical parts with no embedded electronics. The student
creates the program and compiles the code with a portable
scanning station. CyberPlAYce [31] allows the construction
of cyber-physical environments using magnetic modules.
CS Unplugged [4] is an educational method for introduc-
ing Computer Science concepts to non-specialists without
using a computer. The use of materials such as strings or
crayons, help in promoting computational literacy.
Augmented Reality Platforms
AR platforms have been developed to teach concepts from
mathematics [15] to visual arts [7]. The integration with
the environment allows for interactions that can greatly
benefit educational methodologies [18][17]. A number of
these platforms focus on teaching computational concepts
to children and foster creativity. ARScratch [25] adds AR
functionalities to the Scratch [28] platform. In Color Code
[29] children play with physical objects with different col-
ors to program behaviors on the screen. While these two
platforms make use of a computer and a screen, other ap-
proaches take advantage of mobile systems, such as Qbox
[1], AvatAR [8] or Augmented Creativity [37]. More in-depth
systems to teach coding skills such as Code Bits [11] or
PlutoAR [30] make use of paper cards as image targets to
design a program and solve puzzles in AR.
380
Works-in-Progress CHI PLAY'19, October 22–25, 2019, Barcelona, Spain

In contrast, with Paper Cubes [10], CodeCubes [5] and AR-
Maze [14] the user tracks physical cubes instead of cards.
The child plays with digital content around the cubes once
tracked. All these projects make limited use of spatial cog-
nition, requiring the user to remain at the same position and
offer little room to compose in space, with a single view-
point and less bodily engagement. The user is unable to
track objects from more than a meter away, greatly limiting
mobility and usability. Collaboration in real-time and peer
programming become unattainable.
With HyperCubes, we build on top of these systems by al-
lowing the user to move freely in the environment. Hyper-
Cubes anchors the blocks in space in a way where the user
can walk away and the content will stay even if the block
is out of view from the device camera. We lay out a spatial
scenario where we incorporate the concept of body-sintonic
reasoning [23] that refers to the idea of using spatial cog-
nition and one’s own body to plan and execute sequential
commands of an external agent in space.
HyperCubes
Description
HyperCubes is an Augmented Reality platform to play and
tinker with computational thinking concepts by using a set
of paper cubes to create and program digital content in
physical and virtual space. By tracking the cubes with an
AR mobile application, a flow of virtual objects can be con-
trolled. Placing the cubes in different sequential order on
any surface will create different outcomes. Each cube acts
as an input/output module that applies a predefined trans-
formation to the digital flow of elements. The students can
easily build as many cubes as they want with accessible
and cheap material such as paper, scissors and tape.
There are seven types of cubes [Figure 2] with a variety of
predefined actions and effects. Each cube type contains
six behaviors inherently attached on each one of its sides
incorporating options adapted to the ages of late middle
and elementary schoolers [36]. Every time one virtual ob-
ject collides with a cube, the behavior on that cube will be

Spatial cognition for computational literacy

Early AR educational applications focused on subjects
where spatial organization was inherent to the concepts
being taught. In chemistry, AR applications were developed
to learn about the 3D structure of molecules [9]. Later, re-
search has shown how spatial ability plays a critical role in
developing expertise in STEM [34]. Newcombe [22] high-
lighted the importance of spatial thinking for STEM fields,
and how it relates to episodic memory. Children who play
with blocks and puzzles develop a better spatial reason-
ing ability. We build from these theories and from previous
studies [3] which indicated that visuo-spatial memory and
gross motor skills have a positive effect in learning in STEM
fields. We present HyperCubes as a platform to foster com-
putational literacy leveraging spatial cognition.
Figure 3: HyperCubes application interface.
Figure 2: Types of cubes in the
HyperCubes platform
381
Works-in-Progress
Figure 4: Creating a flow of
Characters in Space and setting up
a Checkpoint behavior. From top to
bottom: a. Top view of Emitter and
Transform cube, b. flow of
characters walking on a table, c.
menu to set up a Checkpoint
behavior, d. solving a Checkpoint.
applied to the object. Using the graphical user interface
[Figure 3], the student can tweak parameters and modify
the actions and events applied to each object.
Set of cubes
• The Emitter cube generates the digital objects which
are the base unit of the application.
• The Transform cube performs transformations to the
virtual objects, such as translation, rotation, scale or
color.
• The Split cube creates new instances of the virtual
objects and propagates them in multiple directions,
adding new pathways.
• The Logic cube presents the concept of logic gates
that the student can apply to the virtual objects.
• The Physics cube contains behaviors related to real
world physics concepts such as gravity or elasticity.
• The Effects cube adds visual, sound and animation
effects to the objects and to itself for a playful experi-
ence
• Finally, the Gaming cube contains different elements
that can be used to generate a game scenario. As an
example, the Checkpoint behavior allows the student
to configure conditions that have to be solved with
other cubes.
User Experience
Once a cube is tracked, visual highlighting appears around
the cube. When the student taps on the cube through the
screen on the device [Figure 1], a GUI appears with a side
menu. The parameters of each behavior can be tweaked by
interacting with this interface. Through live experimentation
CHI PLAY'19, October 22–25, 2019, Barcelona, Spain
students learn about the connection between adjusting a
parameter and how it flows through the application.
The loop cycle is initiated with the Play button in the inter-
face and it goes on forever or until the student stops it. The
example in Figure 4 [a,b] shows how the student makes
use of the Emitter cube and the Transform cube to generate
characters and make them move forward in space. When-
ever an object collides with a cube, such as the Transform
cube, the cube lights up to signal its activation. This acti-
vation happens with all the cubes to assist the student in
understanding the sequential nature of their program.
With the Logic cube, the student can make the flow of the
objects asynchronous. Thus, interesting configurations can
give different outputs. For example, using a Sound behav-
ior, a child can create a song.
Another example can be observed in Figure 4 [c,d]. Using
the Gaming cube, the child sets a couple of Checkpoint be-
haviors and creates different conditions. The Checkpoint
being set up expects 10 blue spheres with a scale of two
units. The person trying to solve the puzzle made use of
Portals, Color and Scale behaviors. In the image we can
observe how the first Checkpoint has just been solved, trig-
gering a celebration animation, and the second Checkpoint
has gotten 6 correct virtual objects so far.
System overview and implementation
The software platform used to develop the application is
Unity3D. The AR content is tracked and displayed using the
Vuforia library. The design of the patterns for each cube
is based on a master pattern that is randomized for each
side of the cube and used as a frame surrounding the main
glyph. The main graphical user interface consists of a menu
on the side with a low opacity background. After design it-
erations and feedback from the pilot studies, we decided to
382
Works-in-Progress
Figure 5: Using different cubes to
change behavior and appearance.
Figure 6: HyperCubes emulator.
keep the buttons close to the edges of the device. By keep-
ing controls near the user’s fingers natural resting place, the
controls are easier to adjust while holding on to the device.
To aid in the prototyping and development a desktop emu-
lator was created [Figure 6]. It is an application to configure
and test the cubes on a regular computer. The cubes are
created in the virtual environment mimicking the laws of the
physical setting when getting detected by the AR applica-
tion. The emulator allows the user to load and save scenes.
AR Technologies
The Vuforia library enables us to detect the markers in the
physical environment. It performs the tracking of the cubes.
Additionally, Surface (or Planar) Tracking detects and maps
flat surfaces such as a table or floor. This technology en-
ables the persistence of content in space and is essential
to the HyperCubes experience. To perform Surface Track-
ing, the ARKit and ARCore libraries are used. These two
libraries are incorporated into the Vuforia library and can be
used in Unity. Thus, the multi-targets can be tracked and
anchored in space. The content becomes persistent.
Evaluation
Two pilot studies were organized with children between 8
and 13 years old. Later on, a qualitative user study was
conducted with children of the same age. The first pilot
study was performed in Leicester, United Kingdom, in the
context of the Leicester Art-AI Festival. Three workshops
were performed on the same day with three groups of chil-
dren from the Sandfield Close School [Figure 7]. The sec-
ond one took place at Parts and Crafts, in Somerville, MA.,
in the context of the Girls Invention Week. Overall, 65 chil-
dren played with the application during the pilot studies.
Children were given paper patterns for the cubes and a
document with an explanation of each behavior. Data was
CHI PLAY'19, October 22–25, 2019, Barcelona, Spain
collected using two different techniques: researcher notes
and feedback forms from teachers. The main indicators
used for the notes were duration of play, complexity of the
cubes configuration, collaboration level, need for external
help and usability issues. Teachers answered an anony-
mous feedback form of 18 questions addressing usability,
engagement and applicability to the K-12 curriculum.
The researcher notes analysis asserts that students played
with the application for more than 15 minutes. The com-
plexity of the cubes configuration ranged between creations
with 2 cubes to creations with 10 cubes. Collaboration was
high: they played in groups, discussing what to do next and
trying to achieve certain goals together. Younger students
(<10) asked questions about behaviors. Older students
(>10) used the documentation provided. Some usability
issues were noted and addressed afterwards.
All teachers agreed that the application could be helpful for
the classroom and several subjects could benefit from using
it. Some of them pointed out the need for more feedback on
the cubes so that the interaction was more intuitive.
After the pilot studies, several improvements based on the
feedback were incorporated into the application. Elements
on the GUI were made larger in size to facilitate the interac-
tion and more visual feedback was added to convey when
the events occurred.
Qualitative User Study
A qualitative user study was performed at the MIT Museum
in Cambridge [Figure 8]. A workshop ran from 10am to 1pm
the 25th of July of 2018. Children and parents could walk
in and engage with the application voluntarily. 20 children
between ages 8 and 14 years old participated in the study.
Consent and assent forms were signed by all participants.
Researcher notes were taken during the activity. The same
383
Works-in-Progress
Figure 7: Sandfield Close students
playing with HyperCubes.
Figure 8: Museum workshop.
Qualitative User Study.
indicators were used for the researcher notes. The analysis
of the notes is summarized as follows:
Children and parents stay in the HyperCubes area for more
than an hour indicating a high level of engagement. The
DIY nature of the application and the paper craft is re-
garded as a fun and engaging process. As seen in the pilot
study, the students play together with the application and
discuss what to do next in order to achieve certain goals.
The children make use of spatial cognition, making the digi-
tal objects go up and playing with portals to spread the con-
tent all over the room. They watch the virtual objects collide
with other children in the room. They also stack them using
other objects that are laying around on the table. Children
make use of space to lay out their cube configurations and
move around the cubes to understand the sequential pro-
gram relative to their location in the room.
The students have less difficulty in using the GUI and in-
teracting with the application. They understand the basic
programming concepts behind the different behaviors and
parameters. They also understand how modifying param-
eters will get them different outputs and they are able to
reason which parameters will get them what they need; for
example, using a cube to change the color of a character
or using the Logic cube to send each object to a different
direction or position. Students understand the sequencing
nature of the virtual objects as they travel through space
and are transformed by cube behaviors. The children ex-
periment with the order of the cubes and stop to reason
with and understand the new output. As they continued
playing they noticeably improved at projecting to the fu-
ture and guessing what would occur when placing a certain
cube in a new configuration. They modify the configuration
in order to get the output they want after reasoning and un-
derstanding what is happening.
CHI PLAY'19, October 22–25, 2019, Barcelona, Spain
Conclusions
We have designed, implemented and evaluated Hyper-
Cubes as an AR application that makes use of physical
paper cubes and leverages spatial cognition with the aim of
giving an introduction to computational thinking concepts in
a creative and playful environment. HyperCubes has been
evaluated with pilot studies and a user study, which indicate
high levels of engagement and interest that the application
generates in children ranging from 8 to 14 years old. The
studies have also shown how HyperCubes holds promise
as a potential learning framework and platform to learn not
only about computational thinking but also about other sub-
jects of the K-12 curriculum.
We have successfully implemented and deployed the ap-
plication in a non-lab setting, where children from different
ages have been able to play with it. We leverage a child’s
naturally developed spatial mental models in order to gen-
erate new constructs drawn from existing ones. We argue
that this approach generates an improved learning environ-
ment over previous computational thinking platforms.
The pilot and qualitative user studies gave us a sense of
the level of engagement and interest that the application
generated, but do not offer conclusive evidence on learning
rates or retention by students. A deeper quantitative user
study should be performed with more time and resources.
In this case, we proceeded with a qualitative exploration
that gave us results to show the engagement levels and
potential of the application as an educational tool. For a
future quantitative user study, we aim at comparing learning
rates on computational concepts in the AR setup and the
emulator setup. Students will engage in an activity using
the HyperCubes emulator on a computer and using the
HyperCubes AR application. This will allow us to validate
the learning rates achieved by the AR platform.
384
Works-in-Progress
REFERENCES
1. Judith Amores, Xavier Benavides, Michel Comın, Anna
Fusté, Pol Pla, and David Miralles. 2012. Smart
Avatars: using avatars to interact with objects. (2012).
2. Alissa N Antle. 2009. LIFELONG INTERACTIONS:
Embodied child computer interaction: why embodiment
matters. interactions 16, 2 (2009), 27–30.
3. Mikkel M Beck, Rune R Lind, Svend S Geertsen,
Christian Ritz, Jesper Lundbye-Jensen, and Jacob
Wienecke. 2016. Motor-enriched learning activities can
improve mathematical performance in preadolescent
children. Frontiers in human neuroscience 10 (2016),
645.
4. Tim Bell, Jason Alexander, Isaac Freeman, and Mick
Grimley. 2009. Computer science unplugged: School
students doing real computing without computers. The
New Zealand Journal of Applied Computing and
Information Technology 13, 1 (2009), 20–29.
5. Barbara Cleto, Joao Moura, Cristina Sylla, and Luis
Ferreira. 2018. CodeCubes - Playing with Cubes and
Learning to Code.
6. Stephen Cooper, Wanda Dann, and Randy Pausch.
2000. Alice: a 3-D tool for introductory programming
concepts. In Journal of Computing Sciences in
Colleges, Vol. 15. Consortium for Computing Sciences
in Colleges, 107–116.
7. Ángela Di Serio, María Blanca Ibáñez, and
Carlos Delgado Kloos. 2013. Impact of an augmented
reality system on students’ motivation for a visual art
course. Computers & Education 68 (2013), 586–596.
8. Adso Fernández-Baena, La Salle-URL, and David
Miralles. 2014. Avatar: Tangible interaction and
CHI PLAY'19, October 22–25, 2019, Barcelona, Spain
augmented reality in character animation. on
Interacting with Smart Objects (2014), 28.
9. Morten Fjeld and Benedikt M Voegtli. 2002. Augmented
chemistry: An interactive educational workbench. In
Mixed and Augmented Reality, 2002. ISMAR 2002.
Proceedings. International Symposium on. IEEE,
259–321.
10. Anna Fusté, Judith Amores, David Ha, Jonas Jongejan,
and Amit Pitaru. 2017. Paper Cubes: Evolving 3D
characters in Augmented Reality using Recurrent
Neural Networks. NIPS.
11. Sidhant Goyal, Rohan S Vijay, Charu Monga, and
Pratul Kalita. 2016. Code Bits: An Inexpensive Tangible
Computational Thinking Toolkit For K-12 Curriculum. In
Proceedings of the TEI’16: Tenth International
Conference on Tangible, Embedded, and Embodied
Interaction. ACM, 441–447.
12. Michael S Horn and Robert JK Jacob. 2007. Designing
tangible programming languages for classroom use. In
Proceedings of the 1st international conference on
Tangible and embedded interaction. ACM, 159–162.
13. Michael S. Horn, Erin Treacy Solovey, R. Jordan
Crouser, and Robert J.K. Jacob. 2009. Comparing the
Use of Tangible and Graphical Programming
Languages for Informal Science Education. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI ’09). ACM, New
York, NY, USA, 975–984.
14. Qiao Jin, Danli Wang, Xiaozhou Deng, Nan Zheng, and
Steve Chiu. 2018. AR-Maze: a tangible programming
tool for children based on AR technology. In
Proceedings of the 17th ACM Conference on
Interaction Design and Children. ACM, 611–616.
385
Works-in-Progress
15. Hannes Kaufmann and Dieter Schmalstieg. 2002.
Mathematics and geometry education with
collaborative augmented reality. In ACM SIGGRAPH
2002 conference abstracts and applications. ACM,
37–41.
16. Alan Kay, Kim Rose, Dan Ingalls, Ted Kaehler, John
Maloney, Scott Wallace, and others. 1997. Etoys &
SimStories. ImagiLearning Internal Document (1997).
17. Mehmet Kesim and Yasin Ozarslan. 2012. Augmented
reality in education: current technologies and the
potential for education. Procedia-Social and Behavioral
Sciences 47 (2012), 297–302.
18. Kangdon Lee. 2012. Augmented reality in education
and training. TechTrends 56, 2 (2012), 13–21.
19. James Lockwood and Aidan Mooney. 2017.
Computational Thinking in Education: Where does it
fit? A systematic literary review. arXiv preprint
arXiv:1703.07659 (2017).
20. Timothy S McNerney. 2004. From turtles to Tangible
Programming Bricks: explorations in physical language
design. Personal and Ubiquitous Computing 8, 5
(2004), 326–337.
21. Nora Newcombe. 2017. The connection between
spatial intelligence and STEM. (2017).
22. Nora S Newcombe. 2010. Picture this: Increasing math
and science learning by improving spatial thinking.
American Educator 34, 2 (2010), 29.
23. Seymour Papert. 1980. Mindstorms: Children,
computers, and powerful ideas. Basic Books, Inc, New
York, NY.
CHI PLAY'19, October 22–25, 2019, Barcelona, Spain
24. Shaileen Crawford Pokress and José Juan Dominguez
Veiga. 2013. MIT App Inventor: Enabling personal
mobile computing. arXiv preprint arXiv:1310.2830
(2013).
25. Iulian Radu and Blair MacIntyre. 2009.
Augmented-reality scratch: a tangible programming
environment for children. In Proceedings of conference
on interaction design for children, Como, Italy.
26. A Rees, Francisco J García-Peñalvo, I Jormanainen, M
Tuul, and D Reimann. 2016. An overview of the most
relevant literature on coding and computational thinking
with emphasis on the relevant issues for teachers.
(2016).
27. Alex Repenning. 1993. Agentsheets: a tool for building
domain-oriented visual programming environments. In
Proceedings of the INTERACT’93 and CHI’93
conference on Human factors in computing systems.
ACM, 142–143.
28. Mitchel Resnick, John Maloney, Andrés
Monroy-Hernández, Natalie Rusk, Evelyn Eastmond,
Karen Brennan, Amon Millner, Eric Rosenbaum, Jay
Silver, Brian Silverman, and others. 2009. Scratch:
programming for all. Commun. ACM 52, 11 (2009),
60–67.
29. Eric Rosenbaum. 2010. Color Code. (2010).
https://www.ericrosenbaum.com/color-code/
30. Shourya Pratap Singh, Ankit Kumar Panda, Susobhit
Panigrahi, and Ajaya Kumar Dash. 2018. PlutoAR: An
Inexpensive, Interactive And Portable Augmented
Reality Based Interpreter For K-10 Curriculum. arXiv
preprint arXiv:1809.00375 (2018).
386
Works-in-Progress
31. Arash Soleimani. 2014. CyberPLAYce: a play space of
creative, intelligent tools promoting personal and
computational expression for early learners. In
Proceedings of the 2014 companion publication on
Designing interactive systems. ACM, 191–194.
32. Hideyuki Suzuki and Hiroshi Kato. 1993. AlgoBlock: a
tangible programming language, a tool for collaborative
learning. In Proceedings of 4th European Logo
Conference. 297–303.
33. Cristina Sylla, Pedro Branco, Clara Coutinho, and
Eduarda Coquet. 2012. TUIs vs. GUIs: comparing the
learning potential with preschoolers. Personal and
Ubiquitous Computing 16, 4 (2012), 421–432.
34. Jonathan Wai, David Lubinski, and Camilla P Benbow.
2009. Spatial ability for STEM domains: Aligning over
50 years of cumulative psychological knowledge
CHI PLAY'19, October 22–25, 2019, Barcelona, Spain
solidifies its importance. Journal of Educational
Psychology 101, 4 (2009), 817.
35. Jeannette M Wing. 2006. Computational thinking.
Commun. ACM 49, 3 (2006), 33–35.
36. Chip Wood. 1997. Yardsticks: Children in the
Classroom Ages 4-14. A Resource for Parents and
Teachers.. ERIC.
37. Fabio Zünd, Mattia Ryffel, Stéphane Magnenat, Alessia
Marra, Maurizio Nitti, Mubbasir Kapadia, Gioacchino
Noris, Kenny Mitchell, Markus Gross, and Robert W
Sumner. 2015. Augmented creativity: bridging the real
and virtual worlds to enhance creative play. In
SIGGRAPH Asia 2015 Mobile Graphics and Interactive
Applications. ACM, 21.
387