26-27 September | Gdansk, Poland

Linking GUI Elements to Tasks – Supporting an Evolutionary Design Process
Andreas Wolff, Peter Forbrig University of Rostock Institute of Computer Science Albert Einstein Str. 21, 18059 Rostock, Germany [rusty|pforbrig]@informatik.uni-rostock.de

Anke Dittmar, Daniel Reichart University of Rostock Institute of Computer Science Albert Einstein Str. 21, 18059 Rostock, Germany [ad|dr007]@informatik.uni-rostock.de
Author Keywords

In this paper we discuss an approach for linking GUI specifications to more abstract dialogue models and supporting an evolutionary design process. These specifications are linked to task models describing behavioural characteristics. First prototypes of interactive systems are interactively generated. The automatically generated XUL specifications are refined using a GUI editor, which allows replacing of user interface elements by other elements or components. Components are predesigned parts of user interfaces, and could optionally be parameterised. This enables us to support GUI design using patterns. For illustration we are going to show a design cycle from task model to abstract user interfaces and finally to a concrete user interface. The design process is supported by patterns. Additionally, a proposal is presented of how to keep connections between concrete user interface (CUI), abstract user interface (AUI) and a task model. These connections can be used to propagate changes in a task model or an AUI only to affected parts of a CUI, instead of recreating the whole CUI after a modification has been done. In this paper we primarily focus on deletion, moving and adding tasks to an AUI and the implication of these procedures to a CUI. More complicated problems like splitting or merging of tasks are not addressed in this paper. The avoiding of unnecessary recreation processes for a CUI can reduce the amount of time and work needed in modeldriven software development.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. TAMODIA’05, September 26–27, 2005, Gdansk, Poland. Copyright 2005 ACM 1-59593-220-8/00/0000…$5.00.

Model-Based Design, Task Models, Object Models, Patterns, GUI, XUL, XIML.
ACM Classification Keywords


With the increasing capabilities of mobile devices modelbased development of software systems becomes more and more popular. It is necessary to design user interfaces in an abstract way because there is a diversity of different mobile platforms with specific features. Most of these platforms have to be supported in an economic way by new interactive applications. Models help to derive specifications of interactive systems, and in particular, of user interfaces.

Transformation by patterns
l e dom ksat

Transformation by patterns
ecaf r et ni resu tcartsba tcartsba


Figure 1 - General view on a transformational model-based development process.

We strongly believe that software engineers and user interface designers have to base their work on the same models. Furthermore, we consider software development as a sequence of transformations of models that is not performed in a fully automated way but supported by humans using interactive tools. Our work is especially focused on methods and tools supporting transformations by patterns.

l edom no itacil ppa


l edom IU

))ng is ed(( sisyla na margaid margaid ssalc

hparg hpa rg golaid

)sisylana( margaid margaid ssalc


l edom tcejbo-.b l e dom eciv ed

l edom r esu r esu


26-27 September | Gdansk, Poland

Initially starting with design patterns for class diagrams we more and more believe in patterns as appropriate idea for reuse of knowledge. In this paper we focus on the user interface development process. We present strategies and appropriate tool support (DiaTask) to derive and link user interface models (dialog graphs), corresponding abstract prototypes (XULspecifications) and task models (see the bottom part of Figure 1). The following paper is structured in such a way that after discussing some related work an example of a development process of a specific user interface is given. Afterwards possible changes within the task model and their consequences are discussed. At the end we give an outlook for further work to be done. RELATED WORK During the previous TAMODIA conference there were two talks referring to the “mapping problem” in user interface design ([2] and [11]). This problem was first mentioned by Puerta and Eisenstein [17]. They stated that the mapping problem is the key problem to make model-based development acceptable for programmers. The mappings mentioned in [17] include only mappings from abstract to concrete models and between models of the same level. No mappings from concrete to abstract models are mentioned in their paper. This was carefully analysed by Clerckx, Luyten and Coninx [2]. They have classified five mechanisms to solve the mapping problem. 1. 2. 3. 4. 5. Model derivation Partial model derivation Model linking Model modification Model update.

The initiative for XIML started in 1999 and was focused on device-independence primarily of mobile devices. XIML is model-based but it needs a specific tool (converter) to create a specific type of user interface. Our tool DiaTask [6] was developed to make use of XIML. Task models, user models, and object models with our metaphor of artefacts and tools are represented as XIML specifications. However, there seems to be no further support for XIML. Still there is a lack of tool support. This is especially true for designing a concrete user interface. That was the reason for our group to look for user interface specifications, which are already supported by tools. We found XUL as a candidate for that. XUL [5,14] was presented in 1999 by the Mozilla project to specify Graphical User Interfaces of the Mozilla-browser in platform-independent matter. XUL allows the specification of interactive objects like buttons, labels, and text fields. We can find these objects in tools for creating GUI’s like Java.AWT and Java.Swing. Based on an existing project for eclipse a GUI editor for XUL was developed [25]. It was built in such a way that co-operation wit task models and generated user interfaces became possible. The following example will demonstrate how this editor can be incorporated into the development process of interactive software.

The software developer John has to develop a system to manage mails. One of the forthcoming users is Susan. She works in a small company producing envelopes. John organizes an interview with Susan about the tasks she wanted to be supported by the mailing system. The result of the interview is represented in the task model of Figure 2.
Initial Task Model

Limbourg and Vanderdonckt [11] address the problem by supporting transformation of abstract models to more concrete ones by graph grammars. The user interface specification is based on UsiXML [23]. “UsiXML (which stands for USer Interface eXtensible Markup Language) is a XML-compliant markup language that describes the UI for multiple contexts of use such as Character User Interfaces (CUIs), Graphical User Interfaces (GUIs), Auditory User Interfaces, and Multi-Modal User Interfaces. In other words, interactive applications with different types of interaction techniques, modalities of use and computing platforms can be described in a way that preserves the design independently from peculiar characteristics of physical computing platform”[23]. It seems to be that UsiXML could be a living standard to express models. It can play the role, which XIML [26] originally wanted to gain.

Figure 2 – Initial Task Model

According to the interview with Susan, a very simple mail application is to be developed. It should be able to manage received mails, i.e. to store them and display at request, and also to send new mail.


26-27 September | Gdansk, Poland

According to the task model of Figure 2 a user may either read his mail, or write a new one. To read his mails, he has to select a specific mail from a list that is generated and presented to him by the application. Once he has selected a mail he gets its content displayed. Select and display are consecutive subtasks of an iterative tasks that can be cancelled at any time. Writing mails is modelled in a similar manner. After a user decides to write a mail he has to enter the iterative task produce mail, where he is requested to compose a new mail and, after finishing this, the application will send it away. This sub-task may also be cancelled at any time.
From Task Models to Dialog Graphs

John is an expert user of the system DiaTask (see e.g.[18],[4]), he decides to develop an abstract dialog graph based on the task model (of Figure 2) that was developed together with Susan. This task model is used as input for the dialog graph editor tool. While there is currently no satisfying way of an automatic generation of dialog graphs, the DiaTask editor can be used to create them manually. At first a designer has to decide, how many views are desired, and whether each of them is modal, single or multiple. Next step is to assign relevant tasks to views. The underlying task model determines the set of tasks, which can be distributed on views. Thereafter the designer has to model transitions between tasks and views. DiaTask does support necessary operations to do this. John decided to use 4 different views. The first view is a start screen, where a user decides whether he wants to read or write a mail. Beside this, a single view exists for reading mails, a multi view for writing a mail and a modal dialog was designed to visualize progress during send operations. John decided that it makes sense to read only one mail at a time (single view) while writing different mails at the same time should be possible (multiple view). John attaches tasks to the views and combines them with the corresponding transitions. For details see Figure 3. The reader might recognize that no task is attached to the fourth view (send progress). It was a technical design decision of John to have this window describing the progress of sending a mail. There was no task in the task model forcing to have this view.

Figure 3 –Dialog graph for sample application From Dialogue Graphs to Abstract Prototypes of User Interfaces

Given a dialog graph DiaTask can generate an initial abstract prototype in a WIMP style, which mainly reflects the navigation structure of the user interface. Windows represent views and the elements of the views are mapped to buttons as to be seen in Figure 4 for the example dialog graph of Figure 3. Other task-element mappings can be realized by applying a different presentation model. The generated AUI is stored in XUL format. For testing purposes it is possible to animate the designed dialog graph. DiaTask uses its generated XUL files to have a graphical representation for each affected task. Beside layout information, each XUL element contains control attributes. Especially of relevance is identification attribute (id) of the associated task. This id is generated by DiaTask while editing the dialog graph and is unique for each task. It allows a trace back of interactions on the final graphical user interface to the corresponding task. The id of a task is especially important if new UI-elements are created. Other attributes are used to track changes in task-view assignments. More details on this matter are discussed in chapter “Moving tasks between views”.

Figure 4 - Pre-Generated Views

An animation of the prototype reveals how transitions are interpreted. Clicking on a button representing the start node of a sequential transition let the active window disappear. The window, which represents the destination view of the


26-27 September | Gdansk, Poland

transition, becomes visible and active. Concurrent transitions allow the corresponding window of the source view to stay visible. The user can activate it at any time. For the example, this means that first window (containing “Read Mails”, “Write Mails”) stays open after pressing the "Write Mails" button. Multiple views go together with concurrent transitions. A generated prototype allows the creation of several instances of the corresponding window. John demonstrates Susan the animated abstract prototype. Together they create some test scenarios and observe how the abstract prototype behaves while executing the scenarios. Susan agrees with the models specified up to now but she expresses her wishes to have a look at the final user interface. John promises that this will be possible soon. John was able to promise more specific user interfaces for the prototype because he was aware of our GUI editor [25] available for this purpose. The features of this editor will be described in the next paragraph.
From Abstract Prototypes to Concrete GUIs

related associations of an element and accordingly keep connections to the task model. John starts the editor XUL-E and replaces buttons in the abstract user interface specification by other more appropriate ui-elements. In view number 2 e.g. the button “List Mails” is replaced by a tree structure and a list box, whereas the button “Display selected Mail” is replaced by three labels and some text displayers. “Select Mail” is exchanged by a “OK_Help_Cancel”-component (see Figure 5).

Following the generation of abstracts user interfaces (AUI) testing the dialog structure; in a next step concrete user interfaces (CUI) are to be designed. This can be supported by a XUL editing tool (XUL-E) [25]. It was developed as a plug-in for the rich-client platform “Eclipse” [8]. For information exchange between the tools XUL-E and DiaTask, a slightly enhanced version of the XUL language is used, which is called XULM. Enhancements include the possibility to • • •
Figure 5 - Designed GUI for Select & Read

store multiple views/windows in one file, define a repository for pre-designed components and define placeholders that refer to these components

The editor XUL-E itself is under continuing development, and is currently capable of editing most layout-affecting XUL elements. XUL-E uses DiaTask’s generated AUI as starting point for layout refinements. It is possible to wrap every view of an AUI into a single XULM-file. Alternatively single views can be edited as separate XUL-files. The basic idea of an integrated editing process, as presented here, is to edit by replacements. To design the user interface of a task, another one can replace its current visualization. A user interface designer has to proceed in the following way. At first he has to select a visualization (graphical element) of a task (e.g. a button), choose “replace”, and select via “drag & drop” a graphical element or a predesigned component, which replaces the original one. To proceed in this way makes it possible to maintain task-

Replacing a single element, like a button for example, by a more complex component, raises the problem of where to attach task-related information to. While it would be possible to actually apply these data to every element of a complex component, this is probably not the desired behaviour. Also it is imaginable that components contain visualization stubs for referencing other than the current task (See also section “adding a task” below). Therefore XULM provides a facility of fine-grained control on this matter. An element inside a pre-designed component may be marked whether it should have task-control-data applied or not. The default assumption is that they have not to be applied (value = “none”). A second option is “implicit”, meaning to apply task-control-data of the replaced element. As a third option the value “explicit” can be defined, meaning that this replacement requires user interaction. It has to be decided, whether to delete this element, as it is currently not necessary or to set manually, which task this element belongs to. In the latter case, a consequence might be to adapt the underlying dialog graph and even the task model. In both cases consistency has to be checked. The tool DiaTask is more appropriate for this purpose than XUL-E.


26-27 September | Gdansk, Poland

For logical and organizational purposes XULM offers a feature to group components into packages. Those packages again can contain packages, creating a hierarchy in this way. Packages are stored in repositories – currently XULM-files – which can be referenced from any XULM file and are dynamically loaded by XUL-E or any other tool using XUL-E’s engine. As each package can contain multiple components, it is conceivable to group different visualizations for the same task(s) into one (sub-) package. To support this approach, one component of a package may be declared as default component. By referencing a package, this default would be used for visualization purposes. The editor is prepared to easily replace components of the same package, but only of the same level of hierarchy, by each other. Thus using a different one than the default component of a package is not a problem. Beside this, explicit referencing of a single component is supported too. Generally a reference is defined in terms of XULM placeholder-elements. Thus XULM provides two ways to adapt a CUI to different contexts-of-use, either by using a different repository or by overriding package defaults.

some difference in the views appearances. That is not a real cutback, because if we would want to display all features we could use any gecko-based browser. The advantage of DiaTask’s simulation engine is that it is already integrated.

Figure 7 - A refined prototype of Fig. 4 in animation mode

Susan is impressed by the prototype demonstrated by John. Looking at the user interface in this concrete version she gets a better impression how the system will work in the future. However, the concrete user interface generates new ideas in her mind. At the moment there is no way to attach files to mails. She never mentioned this whish before. Nevertheless, it is very important to have this feature. She asks John to change the original task model.

Missing functionality has been identified by an animated walk-through. To add this functionality it is required first to change the task model. In a second step the new tasks have to be interlaced into the dialog graph and finally the concrete user interface has to be adapted.
Figure 6 - Package structure of repository use for the example

For the example application, five components were created, which were grouped into two packages. Package tamodia consists of three components, which all include the package default of the tamodia.helper package (see Figure. 6). The view “Select & Read” was designed using components as replacements. A tree and a listbox replaced the button of task „List Mails“. Button “Select Mail” was exchanged by a “OK_Help_Cancel”-component and button “Display Selection” consists now of three labels and some text displayers. Note that sample texts have been integrated for demonstration purposes only (see Figure 5). However, after the redesign of all views, DiaTask can still be used to animate the dialog graph. It is possible to discuss the application based on a concrete design instead of rather abstract – button-represented - tasks. The animation of the dialog graph becomes "more readable" for users and is more appropriate for discussions. In Figure 7 an animation state is shown, where a user writes a mail. The simulation engine of DiaTask is currently restricted to a subset of XUL-E’s capabilities, so there is

John develops a new version of the task mode, by adding a task “Show Attachments”. The result can be seen in Figure 8.

Figure 8 – Enhanced task model

Within the second step John integrates the new task into the dialog graph. He decides to put it into the “Select & Read” view. DiaTask notices the change of this view, and marks


26-27 September | Gdansk, Poland
Moving Tasks between views

the new task as “TaskAdded” within XULM. XUL-E is started to define a design for this task. XUL-E interprets the above-mentioned “TaskAdded” attribute and brings out the prototypical visualization of the new task on a highlighted position above the affected view (See Figure 9).

To fulfil this requirement, no changes in task model are needed. Therefore it remains untouched. It is only necessary to edit the dialog graph. John edits the dialog graph. He splits the view “Select & Read” into two separate views. The first view is “Select” containing only selection-related tasks and the second vie “Read Selected Mail”, a view that displays a mails text and attachments. The result of his editing is presented by Figure 11. A detection of those changes is only possible by DiaTask itself, it has to mark any moved task with an attribute “TaskDeprecated” in its old view and also as “TaskAdded” in its new view. Handling of “TaskAdded” was outlined above; there is currently no difference between adding a completely new task and moving a task from one view to another one (see e.g. Figure 11).

Figure 9 - Editor's visualization of added tasks

John is now requested to incorporate this task into current “Select & Read” view. Using component “tamodia.attachmentselector” and some resizing the view of figure 10 is the result. Note the button “Save” in the attachment display-part. It is defined within component “attachmentselector”, attributed with Task-Id=”explicit”. It is up to a designer to keep it, and connect it to an appropriate task or remove it. Here John decided to keep it for the moment, although he will not create any connection to a specific task.

Figure 11 - Displaying a single mail has been moved into a separate view

As support of “TaskDeprecated” XUL-E displays all affected parts of a view surrounded with a thick red border (see Figure 12). A designer has to remove these marked parts from the view and if necessary to re-layout remaining elements.

Figure 10 – Select & Read UI after integration of attachments

Together with Susan John has once again a look at the animated prototype and the designed models. He recognizes that there might be a better design of the system by moving the task “Display Selection” into an own view, which will be only for viewing a mail content.
Figure 12 – Tasks marked for deletion



26-27 September | Gdansk, Poland

Tools for model-based software development have advanced to a development stage at which applying them to real projects may add up to real benefits. We presented strategies and tool support for designing interactive systems, and in particular, user interfaces mainly on the basis of task models. One can consider our approach as complementary to procedures as proposed by TERESA [21] and DynaMoAID [2]. While these references prefer an automatic generation of user interface prototypes from task models, we focus on a transformation-based development process that is interactively controlled by designers. Therefore our strategy is to leave explicit design decisions to interface designers. In the future both approaches could be combined. The tools we presented currently provide support for the animation of abstract canonical prototypes and carefully designed user interfaces. By starting from an abstract user interface a prototype evolves to his final appearance by designing step by step each view of the navigation dialog. Our tools are also able to animate different navigation graphs concurrently. Thus it is possible to look at different design decisions and let users compare and discuss varying versions of the user interfaces. Some shortcomings of the current tool set, DiaTask and XUL-E, have already been outlined in the preceding sections. As for example the fact that a UI design for tasks is lost, when it is moved from one view to another one. Subject of discussion is also the way in which added and removed tasks are indicated. Furthermore, XUL-E should be enhanced to restrict replacements on sensible elements or components only. It has been shown that, in the presented process, DiaTask and XUL-E are closely related to each other. While both are already developed as plug-ins for “Eclipse” there is still some work to do to improve the actual interaction between both, in particular in the XUL-E to DiaTask direction. It might be worth to do some investigation on ways to associate sub-task models to pre-designed components. So a component, or at a lower level of detail a package, could bring along its own task-model, possibly including a dialog graph, and if the component is applied to some view, taskmodel and dialog graph of the application are enhanced by those of the component. This could have been used for the save-Button (see Figure 8) of our sample application. In this case it would contain a small task model for saving attachments. Above all, there is a need to identify patterns that help to transform task models into dialog graphs. As a first step it might be of some help to automatically map specific temporal relations to transitions. For further integration of ui-patterns, it might be necessary to enhance XUL-E with a transition mechanism between

Figure 13 – Newly designed views Deleting a Task

Deletion of a task is essentially the same as the “TaskDeprecated”-part from moving a task. This kind of dialog graph modification can be detected by DiaTask only. Indeed, it is the only case where a change can be made at first in XUL-E. DiaTask detects these forms of changes and propagates them to the dialog graph.

The outlined approach also enables us to make use of uipatterns in interface design. Instances of patterns play the role of components. To use patterns, one would have to design components that contain arranged pattern instances with layout information. Elements within these components are to be marked as Task-Id=”implicit”. Replacing parts of these components with another taskannotated element results in a view, where on the one hand each part of an instance of a pattern has its task assigned. On the other hand a task is related to a pattern and visualized by a pattern instance. This problem of representing patterns by component is the object of undergoing research. Especially the application of collections of patterns in real life projects has to prove the advantages of the outlined approach.


26-27 September | Gdansk, Poland

components that contain more than a single task. It might be located at package level and describe transitions between whose (sub-) components in XULM format.

13. Luyten, K., Clerckx, T., Coninx, K., Vanderdonckt, J.: Derivation of a dialog model from a task model by activity chain extraction. In Jorge, J., Nunes, N.J., e Cunha, J.F. (ed.), Proc. of DSV-IS 2003, LNCS 2844, Springer, 2003. 14. Mozilla.org: XUL Programmer’s Reference 2001. 15. Paterno, F.; Mancini, C.; Meniconi, S: ConcurTaskTrees: A Diagrammatic Notatiob for Specifying Task Models, Proc. Interact 97, Sydney, Chapman & Hall, p362-369, 1997. 16. Paterno, F., Santoro, C.: One Model, Many Interfaces. In Proc. of the Fourth International Conference on Computer-Aided Design of User Interfaces, p. 143-154. Kluwer Academics Publishers, 2002. 17. Puerta, A.R. and Eisenstein, J. Towards a General Computational Framework for Model-Based Interface Development Systems. Proc. of the 4th ACM Conf. On Intelligent User Interfaces IUI’99 (Los Angeles, 5-8 January 1999). ACM Press, New York (1999), 171–178 18. Reichart, D.; Forbrig, P.; Dittmar, A.: Task Models as Basis for Requirements Engineering and Software Execution, Proc. Of. Tamodia 2004, p. 51-58 19. Sinnig, D., Gaffar, A., Reichart, D., Forbrig, P., Seffah, A.: Patterns in Model-Based Engineering, Proc. of CADUI 2004, Madeira, 2004. 20. Teuber, C.; Forbrig, P.: Modeling Patterns for Task Models, Proc. Of Tamodia 2004, p. 91-98. 21. TERESA: http://giove.cnuce.cnr.it/teresa.html

1. Cameleon: http://giove.cnuce.cnr.it/cameleon.html.

2. Clerxkx, T.; Luyten K.; Conix, K.: The Mapping Problem Back and Forth: Customizing Dynamic Models while preserving Consitency, Proc. TAMODIA 2004, P. 33-42. 3. Constantine L.L: Canonical Abstract Prototypes for Abstract Visual and Interaction Design, in Jorge J. A. et. al (Eds): Proceedings DSV-IS 2003, LNCS 2844, Springer Verlag, Berlin, 2003, P. 1-15. 4. CTTE: The ConcurTaskTree http://giove.cnuce.cnr.it/ctte.html. Environment.

5. Deakin, N.: XUL Tutorial. XUL Planet. 2000. 6. Dittmar, A., Forbrig, P.: The Influence of Improved Task Models on Dialogues. Proc. of CADUI 2004, Madeira, 2004. 7. Dittmar, A., Forbrig, P., Heftberger, S., Stary, C.: Tool Support for Task Modelling – A Constructive Exploration. Proc. EHCI-DSVIS’04, 2004. 8. Dittmar, A., Forbrig, P., Reichart, D.: Model-based Development of Nomadic Applications. In Proc. of 4th International Workshop on Mobile Computing, Rostock, Germany, 2003. 9. Eclipse: http://www.eclipse.org.

10. Elwert, T., Schlungbaum, E.: Dialogue Graphs – A Formal and Visual Specification Technique for Dialogue Modelling. In Siddiqi, J.I., Roast, C.R. (ed.) Formal Aspects of the Human Computer Interface, Springer Verlag, 1996. 11. Limbourg, Q., Vanderdonckt, J.: Addressing the Mapping Problem in User Interface Design with USIXML, Proc TAMODIA 2004, Prague, P. 155-164 12. López-Jaquero, V.; Montero, F. ; Molina, J.,P.; González, P.: A Seamless Development Process of Adaptive User Interfaces Explicitly Based on Usability Properties, Proc. EHCI-DSVIS’04, p. 372-389, 2004.

22. UIML Tutorial, http://www.harmonia.com
23. UsiXML: http://www.usixml.org/

24. Wilson, S.; Johnson, P.: Bridging the generation gap: From work tasks to user interface design, In Vanderdonckt, J. (Ed.), Proc. of CADUI 96, Presses Universitaires de Namur, 199, p. 77-94. 25. Wolff, Andreas, Ein Konzept zur Integration von Aufgabenmodellen in das GUI-Design , Master Thesis, University of Rostock, 2004.
26. XIML: http://www.ximl.org