Adaptive Visual Dialog for Intelligent

Tutoring Systems

Jae-wook Ahn(B) , Maria Chang, Patrick Watson, Ravi Tejwani,

Sharad Sundararajan, Tamer Abuelsaad, and Srijith Prabhu

IBM T.J. Watson Research Center, 1101 Kitchawan Rd,

Yorktown Heights, NY 10598, USA
{jaewook.ahn,pwatson,rtejwan,sharads,tamera,snprabhu}@us.ibm.com,
Maria.Chang@ibm.com

Abstract. Conversational dialog systems are well known to be an effec-

tive tool for learning. Modern approaches to natural language processing
and machine learning have enabled various enhancements to conversa-
tional systems but they mostly rely on text- or speech-only interactions,
which puts limits on how learners can express and explore their knowl-
edge. We introduce a novel method that addresses such limitations by
adopting a visualization that is coordinated with a text-based conversa-
tional interface. This allows learners to seamlessly perceive and express
knowledge through language and visual representations.

Keywords: Intelligent tutoring system

Conversational dialog systems · Adaptive visualization · Visual dialog

1 Introduction
Conversational dialog systems allow people to communicate with intelligent soft-
ware in a natural way. Natural user interfaces equipped with conversation abil-
ities, voice recognition, and speech generation have been recognized as a future
user interface in various domains [12] and are already being commercialized.
Such conversational interfaces are useful in intelligent tutoring systems (ITS)
[4], where mixed-initiative dialogues are commonly used to teach conceptual
information [10,21]. They leverage the flexibility and expressiveness of the nat-
ural language, allowing learners to convey partial knowledge and ask questions.
However, we still need to investigate what the best method is to implement effec-
tive conversational interfaces for intelligent tutoring [8], particularly considering
the limitations of natural language interfaces (NLI) [19]. Some information is
better conveyed via visual representations and concept maps are widely used for
learners to visualize relationships and hierarchical organization of ideas.
The interface presented here is a hybrid of two approaches: textual conver-
sation and visualization. It provides learners with a wide degree of flexibility in
reporting knowledge and receiving feedback, but it also scaffolds learner behav-
ior through automated assessment and feedback. The visualization part of the
c Springer International Publishing AG, part of Springer Nature 2018
C. Penstein Rosé et al. (Eds.): AIED 2018, LNAI 10948, pp. 413–418, 2018.
https://doi.org/10.1007/978-3-319-93846-2_77
414 J. Ahn et al.

interface adapts its topological structure and interaction mechanisms based on

the changes in the text-based tutoring conversation flow. This Adaptive Visual
Dialog provides learners with opportunities to interact with tutoring systems in
novel ways that combine NLI and visualizations.
Previous related attempts are categorized into two groups: ITS and adap-
tive visualization. There are several studies that visualize conversations such
as [20] but they do not support dynamic visual interactions between users or
intelligent agents. Many ITS have been developed that use conversational dialog
[18]. Concept map building has been explored in ITS as a method for modeling
complex systems [6] and for learning via teachable agents [14]. Adaptive visu-
alization adapts its visual representations depending on various user features
that are explicitly provided or inferred from the trace of user actions [2], using
one or more adaptation strategies: (a) visualization method adaptation [11], (b)
visual structure adaptation [15], (c) adaptive annotation [16], or (d) open user
(learner) models [3,5]. The current work belongs to categories (b), (c), and (d).
The visual concept map adapts its structure according to the progress of text-
based tutoring conversation and users’ direct manipulation of the visualization.
Specific concepts are visually annotated and dialog states and learner model
metrics are dynamically updated. The changes in learner models cause changes
in the visualization and the user is provided with greater transparency of their
estimated learning state.

2 Adaptive Visual Dialog System

Figure 1 shows a prototype implementation of Adaptive Visual Dialog. It was

designed to improve traditional text-based tutoring systems by blending an NLI
with a visualization. It guides students through conversations by asking ques-
tions, evaluating student answers, eliciting concepts via hints, telling assertions,
answering questions raised by the student, etc. We used IBM’s Watson Con-
versation API [1] to implement the mixed initiative dialog. The conversation
is made not just through the textual chatting box (Fig. 1(a)) but in a tight
connection with an interactive visualization (b). Watson Conversation provides
RESTful APIs that bridges the conversations between a learner and Watson,
so the front-end could easily watch or intervene them through the visualization.
The visualization uses a force-directed network layout [13]. The circular nodes
represent key concepts appearing in the learning material. For example, in the
visualization based on a chapter about “Earth, the Moon, and the Sun” in an
Earth Science textbook, nodes represent sub-concepts (e.g., gravity) and links
between nodes represent latent relationships between those concepts. The visu-
alization allows users to freely explore and comprehend the knowledge structure.
On top of the force-directed network, convex-hulls (colored areas) are drawn to
indicate the corresponding group of concepts are interconnected to each other.
The infomap community detection algorithm [17] was used for fast and reliable
online clustering. By visually examining the communities and the included con-
cepts, users instantly understand how they are co-related and achieve a specific
 Adaptive Visual Dialog for Intelligent Tutoring Systems 415

learning objective (e.g., answering a question). Along with the adaptation fol-
lowing the interactive dialogue, an open learner model [3,5] based adaptation
is supported. The prototype is equipped with a learner model that tracks stu-
dents’ mastery and consistency on concepts. The scores are overlaid on concepts
using two black and white “arcs” that transparently show the user’s mastery
and consistency of the concepts.

Fig. 1. Adaptive Visual Dialog prototype shows a textual dialog (a) and a visualization
that presents concepts and clusters (b) synced with the conversation context. Student
directly asks a question from visualization (c). Mastery and consistency scores loaded
from an open learner model are overlaid on concepts as black and white arcs (d). (Color
figure online)

Interaction Use Case Scenario. When a session begins, a student is asked

a question about a topic. The concept visualization is activated and shows key
concepts and clusters. She responds with an answer and the dialog engine assesses
it and activates a visual remediation step if it is incorrect. Figure 1 depicts that
the student gave a wrong answer to a question. In order to guide them to the
correct answer, the system highlights the most relevant concept node (red) and
its cluster (light blue border) (b). The learner may visually examine the concept
and the other neighboring concepts within the cluster, and attempt the tutor-
initiated question again. If they are still unsure about it, they may click on a
concept to reveal a list of candidate questions about the concept (c). Clicking
on a question causes it to be passed to the text-based dialog. The learner can
ask the tutor the question as is, or modify it in the text entry field. This helps
to the fact that sometimes the learner might not know the right set of questions
to ask on a particular topic during the conversation. The system answers them
by retrieving information from the textbook on which it has been trained. This
enables a type of textual-visual coordination where a user expresses their intent
416 J. Ahn et al.

to the system. We have devised a preliminary set of strategies for how concept
graphs and text-based dialog interactions should be coordinated (Table 1).

Table 1. Textual-visual coordination: dialog-visualization events

Conversational event (Cause) Graph event (Effect)

Tutor initiated question or
suggested topic
Tutor compares contrasts
Tutor provides examples
Graph event (Effect)
Student creates incorrect links
and/or graphical misconception
Student browses nodes
Highlight or glow relevant concepts
Align concepts spatially
Expand from concept node; illustrate that this
is like an instance rather than a new concept
Conversational event (Cause)
Remediation options: (1) directly critique
student graph, (2) spatially align with correct
graph, (3) correct (spatially transform) student
graph
Display links to FAQs and/or examples

Users can also create their visual representations attuned to their own under-
standing of the text. Visual nodes are derived from the annotations made by the
users within the text using the user interface – to select text and drag it to
the visual dialog interface where it is rendered as a new node. Users can then
interact with this node and define relationships in the same manner as they
would with the other nodes in the visual dialog (i.e., it may be assigned to clus-
ters, explored, or linked via edges to other nodes). The very process of creating
and manipulating such visual representations can be thought of as a form of
self-explanation which can be learning activity [7]. These visual representations
can be compared to the standard (i.e. automatically extracted) visualization
via partial graph-matching algorithms [9], enabling concrete feedback on user-
generated visualization. These annotated visual dialogs can also be shared by
the users (students) with their peers and instructors and receive feedback.

3 Conclusions and Future Work

This paper describes our approach for building an adaptive visual dialog for
intelligent tutoring systems, supporting adaptivity with respect (1) to the state
of the conversation and the topic and (2) to learner models. Our prototype
opens the door for learning experiences that take advantage of the benefits of
mixed-initiative dialogues and concept mapping. It also enables learners to have
a more varied experience that recruits spatial reasoning. We hypothesize that
learning experiences can be further improved by selecting topics, visual, and
textual events that our models predict will result in the greatest learning gains.
We plan to test the prototype with real students and we are interested in using
 Adaptive Visual Dialog for Intelligent Tutoring Systems 417

an iterative design approach so that we can discover what textual-visual coordi-

nation strategies lead to greater learning gains and if some strategies are more
effective for some students than others.

References
1. Watson Conversation (2018). https://www.ibm.com/watson/services/conversation
2. Ahn, J., Brusilovsky, P.: Adaptive visualization for exploratory information
retrieval. Inf. Process. Manag. 49(5), 1139–1164 (2013)
3. Ahn, J., Brusilovsky, P., Grady, J., He, D., Syn, S.Y.: Open user profiles for adap-
tive news systems: help or harm? In: WWW 2007: Proceedings of the 16th Interna-
tional Conference on World Wide Web, pp. 11–20. ACM Press, New York (2007).
https://doi.org/10.1145/1242572.1242575
4. Anderson, J.R., Boyle, C.F., Reiser, B.J.: Intelligent tutoring systems. Science
(Washington) 228(4698), 456–462 (1985)
5. Bakalov, F., Meurs, M.J., König-Ries, B., Sateli, B., Witte, R., Butler, G., Tsang,
A.: An approach to controlling user models and personalization effects in recom-
mender systems. In: Proceedings of the 2013 International Conference on Intelligent
User Interfaces, IUI 2013, pp. 49–56. ACM, New York (2013)
6. Bredeweg, B., Forbus, K.D.: Qualitative modeling in education. AI Mag. 24(4), 35
(2003)
7. Chi, M.T., De Leeuw, N., Chiu, M.H., LaVancher, C.: Eliciting self-explanations
improves understanding. Cognit. Sci. 18(3), 439–477 (1994)
8. Coetzee, D., Fox, A., Hearst, M.A., Hartmann, B.: Chatrooms in moocs: all talk
and no action. In: Proceedings of the First ACM Conference on Learning @ Scale
Conference, L@S 2014, pp. 127–136. ACM, New York (2014)
9. Gold, S., Rangarajan, A.: A graduated assignment algorithm for graph matching.
IEEE Trans. Pattern Anal. Mach. Intell. 18(4), 377–388 (1996)
10. Graesser, A.C., Chipman, P., Haynes, B.C., Olney, A.: Autotutor: an intelligent
tutoring system with mixed-initiative dialogue. IEEE Trans. Educ. 48(4), 612–618
(2005)
11. Grawemeyer, B., Cox, R.: A bayesian approach to modelling users’information
display preferences. User Model. 2005, 225–230 (2005)
12. Hearst, M.A.: ‘Natural’ search user interfaces. Commun. ACM 54, 60–67 (2011)
13. Jacomy, M., Heymann, S., Venturini, T., Bastian, M.: Forceatlas2, a continu-
ous graph layout algorithm for handy network visualization. Medialab center of
research 560 (2011)
14. Leelawong, K., Biswas, G.: Designing learning by teaching agents: the betty’s brain
system. Int. J. Artif. Intell. Educ. 18(3), 181–208 (2008)
15. Lehmann, S., Schwanecke, U., Dörner, R.: Interactive visualization for opportunis-
tic exploration of large document collections. Inf. Syst. 35(2), 260–269 (2010)
16. Leuski, A., Allan, J.: Interactive information retrieval using clustering and spatial
proximity. User Model. User Adapt. Interact. 14(2), 259–288 (2004)
17. Rosvall, M., Bergstrom, C.T.: Maps of random walks on complex networks reveal
community structure. Proc. Natl. Acad. Sci. 105(4), 1118–1123 (2008)
18. Rus, V., DMello, S., Hu, X., Graesser, A.: Recent advances in conversational intel-
ligent tutoring systems. AI Mag. 34(3), 42–54 (2013)
19. Shneiderman, B.: A taxonomy and rule base for the selection of interaction styles.
In: Shackle, B., Richardson, S.J. (eds.) Human Factors for Informatics Usability,
pp. 325–342. Cambridge University Press, Cambridge (1991)
418 J. Ahn et al.

20. Viegas, F., Smith, M.: Newsgroup crowds and authorlines: visualizing the activity
of individuals in conversational cyberspaces. In: Proceedings of the 37th Annual
Hawaii International Conference on System Sciences, 2004, 10 pp., January 2004
21. Woolf, B.P.: Building Intelligent Interactive Tutors: Student-Centered Strategies
for Revolutionizing e-learning. Morgan Kaufmann, Boston (2010)