You are on page 1of 36

Comparing Compatible Semiotic Perspectives for the Analysis of Interactive Media Devices

Shaleph ONeill Interactive Design Lab School of Design/Division of Applied Computing Universit of Dundee s!"!oneill#dundee!ac!u$


%he purpose of this paper is to e&plore e&isting semiotic techni'ues in order to identif their strengths and (ea$nesses in anal sing interactive media s stems! %hree individual studies are compared using variations of product semiotics) visual semiotics and *cos revised +, -+at. / ,odor0 model! %a$en individuall each stud does not provide a (holl satisfactor solution to the problems of evaluation! 1o(ever) (hen considered together) the possibilit of an integrated semiotic theor becomes an attractive proposition as an evaluation method! %his paper suggests that older semiotic approaches) (hile useful) are not enough in themselves! In order to be useful to 1CI -1uman Computer Interaction0) the relevant aspects of semiotic theor must be integrated (ith an understanding of interactive interpretation) in such a (a ) as to produce a semiotics of ne( interactive media that is capable of articulating its specific characteristics!

1 Introduction

Semiotics) the stud of sign s stems) is a promising candidate for helping us to understand cognitive ergonomics! %he concerns of both cognition and culture are central to semiotic anal sis) (ith the interface being seen as a message sent from designer to user -de Sou.a) 23345 de Sou.a) 46670! %he interface to a product must

reveal (hat it does and ho( it does things through the collection of ph sical and perceptual characteristics and behaviours that constitute the user interface! Of course ho( people interpret these things depends heavil not "ust on the appearance of the device) but also on the bac$ground of the observers and the conte&t (ithin (hich the interaction ta$es place! %his paper attempts to address the problem of appl ing semiotic theor to the practicalities of interacting (ith interactive media artefacts! 8obile phones have been chosen here because the sit some(here bet(een traditional 1CI artefacts -i!e! tas$ based interaction0 and ne( media -culturall diverse and convergent media appliances0! 9hile these particular phones are no( 'uite old) as the forerunners of a ne( (ave of convergent media technolog ) the still e&hibit the $ind of digital abstraction and media convergence that is becoming increasingl evident in ne( media artefacts! %he are interactive sign s stems) the interfaces of (hich must be interpreted in order to operate them! %he aim here is to appl semiotic approaches from older media domains to an investigation of these $inds of s stems) in order to highlight (hich aspects of semiotic theor are appropriate for developing a broad) culturall informed) critical approach to interface design and evaluation!

AS/SA n: 4;) p! 2<

Current Formal Methods

Cognitive ergonomics ta$es the vie( of 1CI) according to de 1ann -23330) as =a matter of $no(ledge representation and information processing! %he current development of user>centred design methodologies such as *%A? based design -de 1aan) 466@) 466A0 and Displa >Based 1CI -+ita"ima and Colson) 4662) 466<0 as formal methods) focus largel on the use of $no(ledge to operate computers based on the e&ecution of tas$s to achieve goals in an action/evaluation c cle -Norman) 46DD0! *%A? is a formal notation s stem that is used in line (ith an all>encompassing vie( of the design life c cle from a user perspective! Displa >based 1CI focuses on the mental modelling process of the users action/evaluation c cle at the interface) offering a different perspective on ho( mental models (or$! Other standard approaches to 1CI evaluation include) user observation studies) ?O8S models and cognitive architectures -+ata"ima and Colson) 466A5 B rne) 23375 +ieras) 23370! 9hile all of these approaches are valid and in man cases ver successful) the suffer from t(o main problems! ,irstl ) approaches based on user observations and on cognitive ps cholog ) (hile scientificall sound) can ta$e a ver long time to develop in relation to s stem design and evaluation5 this is not al(a s cost effective -B rne) 23370! Secondl ) standard approaches) particularl cognitive models) b their ver nature re'uire to be constrained to specific aspects of interaction! %his limits the reusabilit of such methods) (here cognitive models have to be built for each interactive situation) before the s stem can be thoroughl evaluated -B rne) 23370! As

1CI moves further into e&ploring interactive technolog (ithin the home and leisure applications) models must either become more detailed or 'uic$er alternatives must be sought that concentrate on the broad aspects of these ne( t pes of interaction -+arat) 23375 van der Eeer and del Carmen Cuerta 8elgui.o) 23370! Semiotics does not dispute the need for an understanding of the cognitive processes involved in interaction! Indeed) concepts such as) Normans action/evaluation c cle remain relevant to research in this area) arguabl because semiosis is dependent on some sort of perceptual process -*co) 23330! 1o(ever) in terms of $no(ledge representation and sense ma$ing semiotics brings (ith it ne( (a s (ith (hich to understand ho( interfaces are interpreted during interaction! 9here *%A? is unable to describe the =presentation interface -de 1aan) 466@0 a semiotic approach offers a method directl related to the s ntagmatic structuring of signs and their semantic content! 9hen one e&amines the notions of Displa >based 1CI in semiotic terms one begins to see stri$ing similarities bet(een =associative net(or$s -+ita"ima and Colson) 46620 and the (ider ranging notion of =semiosis in semiotics -*co) 46A;0!


Semiotic heory and Interactive Systems

Semiotics concern (ith the nature and use of signs is a good place to approach interactive s stems from) because some of the central concerns of 1CI parallel those alread present in semiotics! %he notion of the sender and reader in semiotics is not dissimilar to the notion of designer and user) or s stem and user) in 1CI! %a$ing the vie( that computers are machines built on signification and code 8ihai Nadin points out that) =One cannot not interactFone cannot avoid semiotics -Nadin) 23340! %he (hole process of interaction can be seen as an act of manipulating and understanding the signs in an interface! Current strands of research being developed (ithin the 1CI/semiotics communit consist of an approach) based on the classic structuralist concerns of Ceter Bogh Andersen -Andersen) 4663) 466605 radical reassessments of paradigms such as Navigation -Ben on) 233305 the development of soft(are -Nadin) 233305 and in approaches to user centred design in the form of Semiotic *ngineering -de Sou.a) 4667) 23340! Both Semiotic *ngineering -de Sou.a) 46670 and Computer Semiotics -Andersen) 46630 have been around for over a decade (ithout successfull entering into the mainstream of 1CI methods) this ma be due to the diverse nature of the research so far! ONeill and Ben on ta$e these ideas further b e&ploring semiotic theor in relation to the broader cultural issues of interacting (ith ne( media in a range of domains -ONeill) 233<5 ONeill and Ben on) forthcoming5 ONeill and Ben on) 2337a) 2337b5 ONeill et al) 23320! %he research presented in this paper is a continuation of these ideas focusing on a comparison of three different t pes of anal sis) in an attempt to identif the strengths and (ea$nesses of different semiotic approaches! Important issues regarding the development of a semiotic approach to understanding interaction in ne( domains are thus identified!

AS/SA n: 4;) p! 2;


Analysis echni"ues

%he first anal sis is made using an adapted version of Eihmas product semiotics -Eihma) 466<0! Chones are e&cellent e&amples of media that e&ist as products that can be bought =off the shelf) and as such) it (ould seem most appropriate to use Eihmas anal sis techni'ues to establish (hich aspects of semiotic theor can be said to manifest in such products! %he second anal sis is made using +ress and van Leeu(ens visual semiotic method -+ress and van Leeu(en) 466;0! %his has been chosen as a complementar approach to Eihmas) because the operation of mobile phones relies heavil on visual representation and +ress and van Leeu(ens method provides the most in>depth anal sis of visual grammar! %he third and final anal sis is made using Umberto *cos modified +, model -*co) 46A;0) (here it is emplo ed specificall to focus on the process of interaction (ith the phones interfaces! %his is a much more detailed anal sis) as it considers interaction over time and is a first attempt to understand ho( the structuring of d namic signs affects interpretation! Chotographs of the three phones are given belo( and should be used as a constant reference throughout the anal ses!

Figure 1.1 Three phones: the Panasonic GD35, Nokia 5110, and the Nokia 6150

Product Semiotics

Susan Eihma uses semiotic theor to e&plore the nature of product design in order to develop an anal tic method that is derived from the sign categories of Ceirce -Eihma) 466<0! Eihma sees the form of the ob"ect) i!e! the overall ph sical construction of it) as part of that (hich has been designed and therefore as having been designed b some person) in line (ith its purpose and functionalit ! B e&ploring the structure of products in semiotic terms) Eihma manages to articulate the parts of designed products) (hich can be vie(ed as communicative of its purpose and function! In this (a ) she aspects of designed products in functional terms (ith a semiotic frame(or$ that sees a designed product as a bundle of concurrent messages or te&t) not unli$e Barthes anal sis of ne(spaper articles -Barthes) 46AA0! %he anal sis method emplo ed here is based on Eihmas product semiotics) (hich has been adapted and tested b Bosse 9esterlund in relation to anal sing (ebsites -9esterlund) 23320! *ssentiall ) the techni'ue proposed b 9esterlund involves t(o distinct stages! %he first is an anal sis of functionalit ) (hich is emplo ed to discern) (hat a product does) or in the case of design) establish (hat a product should do! %he second stage is to emplo Eihmas classification of sign t pes) and their various elements) in anal sing the product! %he concern in this stud is to focus onl on the second stage) as it is onl this stage that is inherentl semiotic! Belo( is a table -%able 2!40 that describes each of Eihmas sign t pes along (ith the characteristics that are important to anal sing product design! It should be noted here that Eihmas sign t pes are based on Ceirces conception of the sign and his subse'uent identification of icons) indices and s mbols!

AS/SA n: 4;) p! 2A

Sign t pe !cons %he tradition of form


Normall used as a reference for the design of ne( product!

Conformit (ith a product tradition and especiall an divergence from it (ill be noted and can function as a sign! Colour 8aterial 8etaphor 8a signif a 'ualit G e!g! (hite can refer to cleanliness! -Connotative0 8a often refer to 'ualit ) e!g! gilding indicates (ealth5 > concrete) emotional coldness! -Connotative0 %he resemblance of a particular ob"ect to some other ob"ect from another domain) often not a designed ob"ect! ,or e&ample the front of a car might resemble a face! %he period st les li$e art nouveau) 46<3s etc!5 moreover geometric classifications li$e HsphericalI vs! Hs'uareI st les! 1ere again) conformance and divergence from (ell>$no(n st les -if an 0 (ill be salient! Some industrial products are designed for a specific environment) e!g! $itchen) bathroom etc!5 others ma have the -false0 appearance of being so designed) e!g! a sports car appearance!

St le


!ndices A pointing form %races of tools Arro(s and pointers are often found on operating buttons of machines5 sometimes the product itself has such a form! Characteristic mar$s from tools used to ma$e the product in manufacturing e!g! the seam on plastic parts from in"ection moulding! Abrasions) dents) fla(s) dirt etc! Just and corrosion! Drops of (ater on the surface of a bottle indicating cool drin$ Often indicate the technical functions of appliances and computers! %he sound of a product in use! %he distinctive smell of certain products e!g! leather! %he feel of a certain material ma indicate 'ualit or b lifting a container ou can find out if it is empt or not! If the are integral parts of the product itself! *!g! the ardstic$ (ith scale and numbers to indicate measurements!

8ar$s of use Other traces Light and sound Noise Smell %ouch ?raphic figures S "#o$s

?raphic s mbols S mbolic colour S mbolic form Cosition and posture 8aterial

Logot pes) on>off buttons) (ashing instructions on te&tiles etc! A red carpet signifies the high societ of fame fortune and ro alt ! -Connotative0 Uniforms often denote t pe of "ob) ran$ etc! Compositional arrangement e!g! closeness) above or belo() etc! %he 'ualit of a certain t pe of material used in dress ma$ing for e&ample ma signif social status or the character of an event!
Ta#$e %.1 &ih"a's ana$ sis (ra"e)ork


Analysing the Phones #ith Product Semiotics

%he process of anal sis is simpl to use Eihmas sign t pe characteristics as a chec$list b (hich to evaluate a product! One simpl moves through the list establishing (hether or not the product displa s an of the characteristics described and notes do(n (hat the are! 9esterlund points out that not all of these elements are useful in relation to the anal sis of (ebsites! Similarl ) not all of these elements are e&pected to be useful in the anal sis of the phones attempted here! 1o(ever) it is e&pected that the phones) as products) should be 'uite susceptible to this t pe of anal sis! In this stud each individual phone is anal sed separatel and comparisons are discussed after(ards! It should be noted that this techni'ue is not considered to be an e&haustive one but one that provides vital semiotic information about products to designers in a standardised format!

AS/SA n: 4;) p! 2D

!.1.1 Analysing the $D%& Sign t pe !cons %he tradition of form %his phone follo(s the form of traditional telephones) mobile and other(ise in that it has the earpiece at the top and mouthpiece at the bottom) although these are not made overtl Description

obvious! Similarl in a traditional mobile phone configuration of it has an aerial top left) a screen and a $e pad! Colour Blac$) in this conte&t is associated (ith the corporate (orld in (hich mobile phones (ere born) connoting a sense of seriousness and professionalism! Jounded curves Again the corporate (orld is associated (ith the mobile phone even although the are no( utterl pervasive in modern societ ! Cointing forms are evident on a centrall located button) indicating four directions of menu navigation! %his phone has a number of ring tones that signif incoming calls and the deliver of te&t messages! As (ell as a vibrate function that performs the same indicative functions silentl ! %here are t(o graphical figures on this phone that behave in an inde&ical (a G the first indicates the amount of po(er that the batter has left and the second indicates ho( good the reception of the phone is! One is a graphical representation of a batter that empties as po(er reduces) the other is a segmented representation of an aerial that reduces or increases as signal strength varies! %here are a number of graphical s mbols evident on this phone! %hese can be bro$en do(n into t(o groups! %hose that belong to the screen and those that belong the buttons! %he screen s mbols generall denote menu choices or the displa of information such as date/time and the input of data such as (riting te&t messages! %he button s mbols generall denote the function of pressing that button) e!g! the phone logo indicates the button to press to call someone) the signs on the $e pad denote (hich numbers appear on screen (hen the are activated! Some of the buttons ho(ever do not clearl denote their function! %he buttons directl belo( the screen and the button stamped (ith the letter c!

St le *nvironment !ndices A pointing form Light and sound

?raphic figures

S "#o$s ?raphic s mbols

Additionall ) there is a logo printed on this phone belo( the screen! S mbolic colour %he central button on the phone is a different colour from all the others reinforcing its significance in the operation of the phone!

Cosition and posture

%he most salient aspect of the phone is obviousl the screen) signif ing its importance to the operation of the phone! %he screen elements are arranged (ith a logo central) batter po(er top left corner) signal strength top right corner and menu options in both bottom corners! Interestingl the biggest button on the interface is positioned centrall on the phone also indicating its importance to the phones operation!

AS/SA n: 4;) p! 26

!.1.! Analysing the 'o(ia &11) Sign t pe !cons %he tradition of form Again this phone follo(s the form of traditional telephones both mobile and other(ise! %he earpiece at the top is made 'uite obvious and is one of the most salient aspects of the phone! %here is no evidence of a mouthpiece at the bottom! In the tradition of mobile phones it has an aerial top right) a screen and a $e pad) as (ell as an on/off s(itch -top right0! Dar$ ?re ) again this colour tends to denote the sobriet of the corporate (orld Jounded curves Again the corporate (orld Description

Colour St le *nvironment !ndices A pointing form

%here are t(o pointing forms situated to the right hand side of the phone) on t(o separate buttons that are above and belo( each other) indicating up and do(n menu navigation! %his phone has a number of ring tones that signif incoming calls and the deliver of te&t messages! As (ell as a vibrate function that performs the same indicative functions silentl ! %here are also t(o graphical figures on this phone that behave in an inde&ical (a G the first indicates the amount of po(er

Light and sound

?raphic figures

that the batter has left and the second indicates ho( good the reception of the phone is! Batter po(er is indicated b a small batter logo (ith segmented bar above it that reduces as batter po(er fades! Similarl signal strength is indicated b an aerial graphic (ith a segmented bar above it (hich alters in relation to signal strength! S "#o$s ?raphic s mbols %here are a number of graphical s mbols evident on this phone! %hese can again be bro$en do(n into t(o groups! %hose that belong to the screen and those that belong the buttons! %he screen s mbols generall denote menu choices or the displa of information such as date/time and the input of data such as (riting te&t messages! %here are t(o buttons on this phone that do not clearl denote their function) one is the central button that has a blue bar across it the other is the button left of it that has the letter c stamped on it! %he $e pad s mbols generall denote (hich numbers appear on screen (hen the are activated! Additionall ) there is a logo printed on this phone above the screen! S mbolic colour %he on/off s(itch is red indicating its importance! %he s mbol on the central button is blue identif ing it as different from all the other buttons! Interestingl the most salient features of this phone) apart from the screen) are the earpiece situated above it and the largest button situated in the middle of the phone! %he screen elements themselves are arranged (ith a logo central) batter po(er to the right) signal strength to the left and menu option belo( centre!

Cosition and posture

AS/SA n: 4;) p! 73

!.1.% Analysing the 'o(ia *1&) Sign t pe !cons %he tradition of form Once again this phone follo(s the form of traditional telephones both mobile and other(ise! %he form of the earpiece differs from the other t(o and is linear rather than round! Again there is no evidence of a mouthpiece at the bottom! It has an aerial top right) a screen and a $e pad) as (ell as an on/off s(itch -top right0! Blac$) again this colour tends to denote the sobriet of the corporate (orld! Jounded curves! Again the corporate (orld! Description

Colour St le *nvironment !ndices A pointing form

%here are t(o pointing forms situated centrall in the phone on t(o separate buttons that are above and belo( each other) indicating up and do(n menu navigation! %his phone has a number of ring tones that signif incoming calls and the deliver of te&t messages! As (ell as a vibrate function that performs the same indicative functions silentl ! %here are also t(o graphical figures on this phone that behave in an inde&ical (a G the first indicates the amount of po(er that the batter has left and the second indicates ho( good the reception of the phone is! %hese signs are absolutel identical to those found in the <443!

Light and sound

?raphic figures

S "#o$s ?raphic s mbols Again there are a number of graphical s mbols evident on this phone) (hich can again be bro$en do(n into t(o groups! %he screen s mbols generall denote menu choices or the displa of information such as date/time and the input of data such as (riting te&t messages! %he $e pad s mbols generall denote (hich numbers appear on screen (hen the are activated! %here are also t(o buttons on this phone that do not clearl denote their function! *ach one is a mirror image of the other and carries the same blue sign as the central button on the <443! %he are also positioned centrall at either side of the

phone in the manor of unidentifiable buttons on the ?D7<! Additionall ) there is a logo printed on this phone above the screen! S mbolic colour Cosition and posture %he on/off button is red indicating its importance! Blue bars on the unidentified buttons indicate their importance! %here are no overtl salient features to this phone as all of the buttons are of a similar si.e! %he screen elements themselves are arranged (ith a logo central) batter po(er to the left) signal strength to the right and menu options in both bottom corners!



%here are man obvious similarities in form and function across these three mobile phones! %he some(hat repetitive nature of each anal sis ma$es this patentl clear! All sho( the same iconic features that are part of the telephone tradition5 the are all similar in colour) the all have earpieces) aerials) screens) and $e pads) (hich are organised in the same sort of configurations! Also) the all e&hibit inde&ical features of remar$able sameness! %he all have pointing forms) ring tones and graphical features that perform the same inde&ical functions) even although their form might var from phone to phone! Lastl ) the s mbolic features of all three phones are again remar$abl similar! %he all e&hibit evidence of similar s mbols in both the screen elements that displa menu choices and information) as (ell as in the buttons of the $e pad!

AS/SA n: 4;) p! 74

9hile the similarities bet(een these phones identif the presence of icons) indices and s mbols) in the interface) it is in the differences bet(een the signifiers and arrangement of signifiers of these three phones that semiotic anal sis comes into its o(n! ,or e&ample) (here one phone signifies its navigational elements (ith a large central button containing four arro(s) the other t(o have onl an up and do(n navigational signification! %he arro(s used to represent these functions also var (hile the function remains the same! Similarl ) the organi.ation of these elements varies from phone to phone (hile functionalit and meaning are maintained! Some signs indicate the same meanings across phones (hile having a different s ntactic structure or position! 1ere (e have found evidence not onl of icons) indices and s mbols but also of the arrangement of signs into concurrent s ntagmatic

relationships that var slightl from phone to phone! %his is also e&hibited on the screens (here batter life and signal strength are displa ed in similar (a s across the phones but occup completel different areas of the screen! In short) the organisation of phone elements) those that follo( a traditional pattern and those that var from phone to phone e&hibit evidence of (hat *co identified as =rhetorical forms -*co) 46D;0 1o(ever) this particular anal sis techni'ue does nothing to engage (ith the problem of interactivit ! It stops short at uncovering the structure and meaning of interface elements and does nothing to articulate ho( meaning unfolds as users interact (ith the phone! 9hile this techni'ue could be e&tended to anal se the signs in each screen during interaction) there is no structure to the approach b (hich to do so) as the techni'ue focuses on identif ing sign t pes rather than configurations of signs!

+isual Semiotics

In Reading Images -+ress / van Leeu(en) 466;0 ?unther +ress and %heo van Leeu(en concern themselves primaril (ith the tas$ of isolating and defining the different methods of construction used in image ma$ing that allo( meaning to be conve ed! %heir in>depth stud of all $inds of images leads them a(a from traditional semiotic evaluation) in the sense of procuring meaning through the relationships bet(een the various signifiers in an image and into a deeper concern (ith the s ntactic construction) or grammar) of images as a (hole! Eisual grammar considers the composition of spatial s ntagms (ith regard to the =informational value of the positioning of elements (ithin an image! ,rom a particularl 9estern perspective) +ress and van Leeu(en propose that the =left side of an image is the =?iven side5 the alread $no(n side5 the start of an idea) as in the headline or opening paragraphs of a maga.ine article for e&ample! %he right side is the =Ne( side) often a photograph) in the case of maga.ines! It usuall demands attention or is problematic in some (a ! %he left to right direction of reading also forms some $ind of narrative that is lin$ed to se'uential s ntagms in a (a ! Obviousl ) this does not appl in cultures (here signs and s mbols are arranged to be read up and do(n or from the bac$ of a boo$ to the front as in Chinese or other Asian cultures! ,or +ress and van Leeu(en) aspects of images that are spatiall organised in the top section of images are considered to be =ideal =good or =(hole) (hile elements that are in the lo(er sections of images are considered to be =Jeal) =Base or generall more do(n to earth! %his is particularl true of paintings that contain religious motifs! ,inall ) (hen a pictorial element is presented in the centre of an image) it is presented as a nucleus of information around (hich all other elements become marginali.ed) subservient or dependent! %hese ideas are closel related to notions about embodied understanding that La$off and Kohnson define in their (or$ on metaphor theor -La$off / Kohnson) 46D3) 46660 (here the consider orientation

metaphors in relation to conceptual understanding of the (orld! %he spatial organi.ation of s ntagms then) derive much of their meaning in relation to bodil understanding and orientation in the (orld!

Figure 3.1 *ress and &an +eeu)en,s &isua$ Gra""ar -*ress and &an +eeu)en, 1..6 p%0/0

AS/SA n: 4;) p! 72

Other aspects of visual structure that +ress and Ean Leeu(en discuss in terms of importance to anal sis are the Salience of ob"ects e!g! si.e) sharpness of focus) tonal contrast) colour contrasts and placement5 framing e!g! the degree b (hich units of information are demarcated as independent from others) and the liner/nonlinear composition of te&ts e!g! the use of subheadings) emphatic devices) numbered lines) tables) diagrams and so on that encourage readers to scan or s$ip read the information instead of reading it in a standard se'uential mode! 1 perte&t is a perfect e&ample of this!


Analysing the Phones #ith +isual Semiotics

%his stud uses the visual semiotic techni'ues as a method for evaluating the three mobile phones! Bearing in mind that +ress and van Leeu(ens theories are geared more to(ards the anal sis of visual surfaces such as paintings and diagrams) rather than mobile phones) it is not e&pected that all of their techni'ues (ill be useful in this anal sis) although some results in certain areas are e&pected to be revealing!

*mplo ing the techni'ues of visual semiotic anal sis involves stud ing the three phones in relation toG narrative processes) conceptual representations) anal tical processes) representations and modalit ) as (ell as informational value and salience! Unli$e the stud above) (here each phone (as treated separatel ) all of the phones are discussed together in relation to these concepts!

%.1.1 'arrative Processes Narrative processes are indicators of visual narrative! +ress and van Leeu(en specificall identif the diagonal tension bet(een organised visual elements to e&press this! On serious contemplation of the surfaces of the phones) it becomes evident that there is no trace of such visual narrative processes! %hat is to sa ) that (ithin the inter>relationships of the components of the interfaces there is no strong diagonal) (hich attempts to conve transactional actions to the observer! Of course) it is evident that there are man different components in the interface) (hich are related to one another in some (a ! 1o(ever) it (ould appear that (ithin their organisational structure there are no stories to be told and therefore no narrative processes! 9here arro(s do appear on the phones) e!g! on some of the buttons) the do not fall into the categor of narrative process as defined b +ress and van Leeu(en) (hich specificall focuses on directional relationships bet(een pictorial elements! %he arro(s on the phones do not point to other elements of the phone directl ! 1o(ever) there are some narrative processes hidden (ithin the functionalit of the phone! ,or instance) the navigational aspects hinted at b the arro( buttons) move the user into an interactive narrative) (hich the construct b interacting (ith the phone! %his narrative thread brings the understanding of interface signs into contact (ith the e&ternal narratives of peoples lives) such as calling a sic$ friend or te&ting someone a football score line! 9hile these ma be considered as goals in themselves) it is interesting to consider ho( such a call or message fits into a (ider meaning>ma$ing conte&t) made available b the mobile phone itself!

%.1.! Conceptual ,epresentations 9ithin all three phones it is apparent that there is an underl ing structure described b +ress and Ean Leeu(en as a Hmultileveled overt ta&onom I! %his is (here elements are organised into a visual hierarchicall ! %he space occupied b the screen in all three phones is the super ordinate element of the ta&onom ) all other elements are therefore subordinate to the screen! %his is emphasised b the screen being placed above the other button elements even although the are related to the operation of the information on the screen! On the ?D7< the large silver button in the centre of the phone) directl beneath the screen) is set above the importance of the smaller operational buttons that occup the rest of the interface! So (e have a three> tier ta&onom ! %his is also similar in the other t(o phones although the si.e and position of the central button varies5 there is still a cluster of buttons that surround a central larger one! %his in itself is another visual hierarchical organi.ation of elements -,igure 7!20!

Figure 3.% 1u$ti$e2e$ ta3ono" in the GD35 phone

%.1.% Analytical Processes It might be argued that the different component parts of the phones are organi.ed through (hat +ress and Ean Leeu(en call =e&haustive anal tic process) i!e! the component parts of the phones ma be organi.ed in such a (a as to ma$e obvious all of their features! %his is true in terms of inde&ical signs) such as the batter life and signal strength icons identified in the previous stud ! 1o(ever) this point of vie( brea$s do(n in terms of interface structure because man of the phones features are hidden a(a from sight) in the memories and central processing units of the phones! %hese are accessed at different times through the screen and are therefore not displa ed in a one to one relationship! %hat is to sa that the component parts of the interface do not correspond directl to the phones features! Instead) it is the combination of spatiall organised signs) coupled (ith the interactive possibilit of restructuring them into various =commands that point to the functionalit of the phone!

AS/SA n: 4;) p! 77

%.1.- ,epresentation . Interaction +ress and Ean Leeu(ens definitions in this section are ta$en largel from the stud of paintings) photographs and %E! Although their ideas of interaction seem to be applicable the ) are couched in terminolog that is entirel inappropriate for

dealing (ith 1uman Computer Interaction) i!e! interaction defined as the active manipulation of a computer interface! %herefore) no significant data about the phones interface can be determined b the application of this particular set of definitions!

%.1.& Modality Similar to the section above the definitions outlined b Leeu(en seem inapplicable in this form of anal sis! +ress and Ean

%.1.* Informational +alue Loo$ing for evidence of a =left/right organisational structure one immediatel notices that this is a prevailing method of organisation across all the phones in a number of (a s! ,irstl ) there are the logos Canasonic) Eodaphone) No$ia etc! above and belo( the screens! As obvious as this ma seem) the positioning of these t(o logos has an important effect! %he positioning of these (ords) ne&t to the most salient component of the interface) ensures ma&imum impact in displa ing the manufacturers and the net(or$ providers names (hereb no interaction) or even cursor glance at the phone) is passed (ithout their authorit ! 8ore subtl though) the left/right direction set up b the logos ne&t to the screens primes users for information delivered in left/right fashion b the screen itself! Secondl ) the +e pad buttons are also numbered from Left to right) as are the letters! %his is of noticeable significance (hen ou ta$e into account the function of the buttons (hen sending =te&t messages Loo$ing for =top to bottom organisational structure (ithin the phones interfaces (hat becomes apparent is the ta&onomic structure as outlined in the conceptual representation section thusG Screen) Control panel) +e pad! Using +ress and Ean Leeu(en definitions the screens are the =Ideal top part of the phones and the +e s are the =Jeal lo(er half of the phones! %his is interesting because the focus of attention is on the screen (hen using a phone) but the $e s) being in the =Jeal section of the phone) are given to the user as tangible) usable artefacts! It should also be noted here that this arrangement has a long histor of development through t pe(riters) %E remote controls) Computers and on into mobile phones as a fairl standard form of representation during te&t manipulation! It also further separates the graphics on screen from the graphics on the buttons! %he buttons are interactive (hile the screen elements are onl there to displa information! Centrall there is the control panel) (ith a central button in all the phones! %he other buttons are out in the margins of the central band! %his ensures the relationship of dominance/subservience outlined earlier in conceptual representations!

%.1./ Salience As noted earlier) the most salient component of the phones interfaces are the screens! %his is because of their si.e -the ta$e up about a third of the available

space0) their position and because of the surrounding facing that stands out against the bod of the phone! In the case of the ?D7< this is silver) in the case of the other t(o the are blac$! %he second most salient components of the phones are the central control buttons! In $eeping (ith the ta&onomic structure outlined earlier) the position si.e and colour of these buttons give them ver important characteristics! %heir centralit means that the are a strong point of focus! 9hen the phone is held read to be used the users thumb sits directl over them suggesting that the might be the most used button on the interface! %heir si.e) usuall the biggest button on the interface) gives them a tremendous pull in terms of usabilit ! %he almost (ant to be pressed! Clearl there is a strong relationship bet(een these buttons and the screen above! %he subordinate function buttons in the mid section of the phone are arranged around the central button and marginali.ed! In terms of salience the seem less significant as a grouping than the $e pad! 1o(ever) their position above the $e pad gives them a sense of hierarchical po(er (ithin the ta&onomic structure! %he are clearl related to the function of the central button! An interesting aspect of this is the difference bet(een the three phones here! On the ?D7< the central button has arro(s on it that seem to denote navigation or direction! Similarl ) the ;4<3 has central buttons that sho( arro(s but onl up and do(n! On the <443 these directional buttons are to the left of a larger central button) still forming a central group that are interrelated! On each phone the screen is situated at the top and belo( are the $e pads! %he s'uare formed b the regimental grouping of the $e s into the ro(s and columns of the $e pad ta$es up a significant portion of the space on the interface! %hus) due to its form) as a group) it has its o(n sense of (eight (ithin the interface structure as a (hole) forming the culturall recognisable unit of the telephone touch> tone $e pad) common to all manner of modern phones! %his last aspect in itself produces a strong sense of salience in that it is a ver recognisable form!

AS/SA n: 4;) p! 7@

Considering the suitabilit of this t pe of anal sis in relation to the mobile phones) it is apparent that there are onl a fe( concepts that are an use in evaluating them! %his is because this techni'ue is derived from concerns (ith the visual aspects of images rather than interactive ob"ects! Nevertheless) a number of concepts (hen applied do provide some data about the semiotic nature of the phones! Avoiding an useable terminolog for categorising sign t pes per se) +ress and van Leeu(ens theories concentrate on uncovering the organisational) or as the term them =grammatical) relationships bet(een visual elements! In this (a ) aspects of the hierarchical relationships bet(een phone elements are identified on t(o occasions! 8oreover) a propensit is identified (ithin all of the phone structures to

promote a left>to>right reading of elements both on the screen and in the organisation of the buttons themselves! %he t(o t pes of organisational structuring identified in this stud then appear to be e'uivalent to the structuring of interface signs as concurrent s ntagms! It is the similarlities bet(een phones that confirm this) (hereb the various forms of elements that differ from phone to phone purport to similar sorts of functionalit across phones! In short) (hile the signifiers alter in loo$ and location across the three phones) their meanings sta the same! It is onl possible to understand this from a semiotic perspective if the interfaces of the phones are considered as a paradigmatic langue) as Saussure (ould describe it -Sassure) 46;;0 of phone signs organised into concurrent s ntagmatic relationships! Again) this t pe of anal sis fails to reall get to grips (ith the process of interaction! 9hile it uncovers hints of it (ithin the signs e!g! the navigational buttons) it does not have a mechanism b (hich to deal (ith the se'uential aspects of interaction anal sis!

0co1s ,evised 2F Model

%he (idened hori.on of social and cultural semiotics developed b Barthes -46AA0 is (hat gives Umberto *co the bac$ground to his unif ing theor of semiotics! Also based on much of the research performed b 1"elmslev -46;40) his Htheor of SemioticsI -*co) 46A;0 is a highl developed re>evaluation of the ma"or branches of semiotics from both the Saussurean and Ceircean schools of thought! *co produces not so much a ne( definition of the sign but a definition of the sign that ta$es into account the m riad social) cultural and conte&tual issues that underlie ever instance of sign use! In doing so) *co proposes a theor of semiotics in terms of the use of signs as acts of coding and decoding messages (ith reference to sets of culturall defined conventions! %he socio>cultural aspects of semiotics and the importance of conte&t in evaluating meaning are central to his theor ! Based on the (or$ of +at. and ,odor) *co develops a d namic model of the semantic aspects of signification that ta$es into account the circumstances and conte&ts on (hich the denotation and connotation of signs are so dependent! *cos conception of signs as aspects of codes) (hich run along and across the various social groups (hich ma$e up societ as a (hole) are based on the notion that for a sign to be understood the reader has to be =in possession of the correct code to interpret it! It is this coding and decoding of signs) (hich *co attempts to model in his revised +at. and ,odor -+,0 model -,igure @!40!

Figure 4.1 The re2ised *at5 and Fodor -*F0 "ode$ -6co, 1.760

AS/SA n: 4;) p! 7<

In e&planationG a sign vehicle /s>v/ is a signifier (hich is formed b a set of s ntactic mar$ers -sm0! %his sign vehicle then has a meaning LLsememeMM that can be either a denotation d or a connotation c depending on the conte&t -other signs (ithin its s stem -cont00 and circumstances -signs outside of its specific s stem NcircO0) (ith (hich it is encountered! %he conte&tual and circumstantial parameters in (hich the sign vehicle is encountered affect the t pe of meaning that the sign vehicle ma pertain to! In other (ords) the denotative and connotative meanings that a sign vehicle might have alter depending on (hen and (here the sign vehicle is encountered -*co) 46A;) p 43<0! ,or e&ample) the (ord =blue might be encountered in relation to =s$ ) =grass and =feeling! *ach alternative (ord alters the meaning of blue offering different denotations and connotations! =Blue s$ simpl denotes the colour of the s$ ! =Blue ?rass is a t pe of American fol$ music! =,eeling Blue connotes an emotional state!

-.1 Analysing the Phones #ith the ,evised 2F Model

%his stud differs from the other t(o in that it is not ta$en from a domain of applied semiotics but relies on *cos theor of semiotics directl ! Li$e the previous t(o studies the aim is to evaluate the elements of the phone interfaces in relation to

the concepts identified as relevant to ne( media! %he premise here is that *cos revised +, model) (here denotations and connotations are dependent on the conte&t and the circumstances of organisational structuring) should be directl applicable to the phone elements as signifiers that bear meaning! %herefore) the established method of anal sis e'uates conte&t -Cont-0 in the diagrams presented belo(0 (ith the se'uence of screens that are displa ed through interaction and circumstance -CirNO0 (ith the concurrent structuring of signs on each screen! In doing so) the revised +, model is adapted to anal sing se'uential and concurrent s ntagms (ithin phone interfaces! It is important to point out here that in the diagrams presented belo( connotations are omitted purel because of the number of images and the space the (ould ta$e up! 9here possible connotations are at least alluded to if not e&plicated in the supporting te&t!


he perceived meanings of phone signs

According to semiotic theor -*co) 46A;5 1"elmslev) 46;;5 Anderson) 4663 0 semantic fields are essentiall the range of possible meanings that are associated (ith an particular sign! In this respect) a semantic field of a specific (ord contains the s non ms of that (ord e!g! beautifulG > good>loo$ing) gorgeous) stunning) attractive! All of these (ords are related in meaning and the use of the sign =beautiful ma refer more closel to one of these other (ords depending on its conte&t of use! In relation to the signs on the phones) it is interesting to e&plore their semantic fields because the are relativel ne( cultural phenomena that consist of ne( and old signs! 9ith this in mind) it is the control section of the phones and the screen that are most interesting! Buttons are designed (ith s mbolic codes that are supposed to communicate their function) (hile screens consist of integrated s mbols that communicate information to the user about the functioning of the phone! On the <443 and the ;4<3 there are t(o buttons one above the other that have arro(s printed on them! On the <443 the are to the right hand side of the phone! On the ;4<3 the are in the centre of the phone! On the ?D7< there is a large central button that has arro(s pointing up) do(n) left and right! %he possible meanings of this sign in relation to different conte&ts are outlined belo( using *cos Jevised +, model -,igure @!20!

Figure 4.% The se"antic (ie$d o( the arro) signs on the GD35

In this diagram the button is presented on the left hand side! %he arro(s point to(ards the conte&tual situations in (hich the sign might be encountered and the subse'uent denotative meanings associated (ith them I!e! the semantic field of the sign! Culturall then) this sign has a number of different meanings associated (ith it that are brought to an interaction! It is not e&actl clear (hat the button is for but the signs on the button denote something about navigation or direction! %he same can be said of the arro( buttons on the t(o other phones -see ,igure 4!40! %his particular sign carries connotations (ith it too) in as much as the directional aspect of its compass form bring to mind the NA%O logo!

AS/SA n: 4;) p! 7;

Similarl on the ?D7< and the <443 there is a button (ith a single =C s mbol printed on it! %he possible meanings of this s mbol outlined as a semantic field is given belo(! Obviousl there ma be other conte&ts in (hich the s mbols in these diagrams might have different meanings! %he e&amples given are purel illustrative and not e&haustive!

Figure 4.3 The Se"antic Fie$d o( 89, on the GD35

,igure @!7 sho(s ho( the different conte&ts that the =C s mbol is used in can offer multiple meanings! It ma purel denote the verbal or (ritten phoneme =C! It ma denote the =clear function on a calculator or it might denote the element of Carbon in chemistr !

Figure 4.4 The e3tended se"antic Fie$d o( 89, on the GD35

In the case of the mobile phones though) it becomes apparent through interaction that =C stands for the =Cancel function -,igure @!@0! %herefore the semantic field for =C is e&tended and (hile it denotes =Cancel in the phones it retains these other possible meanings) the conte&t of the mobile phone interface perhaps connoting meanings associated (ith a $e board or calculator!

-.1.! Analysis of Interaction 3sing 0co1s ,evised 2F Model %a$ing a single s mbol as a starting point an e&ploration can be made) using *cos revised +, model of its functional meaning throughout the process of interaction! %he sign -,igure @!<0 onl appears on the ?D7< (hile buttons (ith a similar position on the ;4<3 have a small blue dash on them) as does the central button under the screen on the <443! It is not at all clear (hat this sign means or indeed if the buttons on the ?D7< perform the same functions as on the other phones!

Figure 4.5 The se"antic Fie$d o( an unkno)n sign

At first glance this s mbol is ver difficult to decipher) as it bears no relation to an of the e&ternal codes used on the phone! %herefore it must be uni'ue to the interface -although similarities ma e&ist in other mobile phone interfaces0! It is constructed from a large rectilinear shape) (hich has a smaller dar$ filled rectilinear shape occup ing its bottom right hand corner! %he problem here is that *cos formula seems largel useless because the sign is so ne(! 1o(ever) if (e follo( his rationale and attempt to shed light on this rectilinear sign b e&ploring the conte&t of its use (e can pic$ up some clues as to its meaning! 9hat needs to be ta$en into account in relation to interactivit is the fact that) as Andersen points out -Andersen) 46630) it is not until a button is pressed that its functional meaning is ultimatel revealed! %herefore) the proof of (hat the sign is attempting to communicate is onl verified through interacting (ith it! An entirel ne( form of semiotic model ma be re'uired to account for this t pe of activit but this does not mean that *cos formula cannot be used at this stage!

AS/SA n: 4;) p! 7A

Cresumabl ) considering the nature of activating buttons) (hen the button is pressed some activit (ill occur (ithin the phones s stem! %his gives us t(o distinct semiotic phases) if (e consider that b pressing this particular button this activit (ill be registered on the screen! So (hat does the change in state sa about the functionalit of the button and does this shed an light on the meaning of the sign printed on the buttonP ?iven the fact that the meaning of a sign in *cos model is entirel reliant on its denotations and connotations in relation to differing conte&tual and circumstantial variables) it follo(s that the comple& alterations in meaning that arise (hen a sign is vie(ed in different situations can be captured b it! %his fle&ibilit in *cos model then results in an opportunit to vie( those! %hat is to sa ) it allo(s the mapping of different stages of meaning that are denoted and connoted b the signs of a s stem (hen the state of that s stem is altered during an interaction! Choosing the interaction goal of =chec$ messages for the stud of the mobile phones limits the number of stages mapped! Concentrating onl on the ?D7< interface) the meanings of the active button signs are mapped here in relation to the circumstances and conte&t denoted b the other signs in the interface! Initiall ) it is not clear as to (hat some of the control buttons signif but through stud ing the interaction (ith the device it becomes apparent that the hold a close relationship (ith the signs displa ed on screen and (ith each other!

Figure 4.6 The 1ain screen o( the GD35 -Note: a$$ diagra"s o"it connotations due to space restrictions0

,igure @!; sho(s (hat the initial screen loo$s li$e at the start of interaction along (ith the corresponding semantic field of the right hand button in relation to the displa ed information! B focusing on the meaning of the active sign at each stage of the interaction) it is possible to see ho( the operation of the device is conve ed through the relationships bet(een its signs! As the interaction ta$es place the signs change and meanings are altered! Sign elements come together over time in order to form se'uential s ntagms that the user interprets as the interaction ta$es place! On the =8ain Screen the function of the right hand button is denoted b NmenuO in the blac$

s'uare at the bottom right of the screen! *cos model sho(s ho( the action of pressing the button results in the function LLgo to menuMM!

UserAction: Cress the right hand button! System State: %he Screen no( sho(s a list of functions (ith an arro(head pointing from left to right at the beginning of the first heading on the screen!

Figure 4.7 1enu screen

,igure @!A sho(s ho( the functional meaning of the right hand button has altered! Cressing the button (ill not result in the same operation because the circumstances have changed! %hese changes are indicated b the presence of a list) an indicator arro( to the left of the screen and N*nableO in the bottom right of the screen! 1ere the interaction moves to a different input button that performs a different function (hich has a meaning all of its o(n! Loo$ing at both buttons (ith *cos model begins to sho( ho( comple& the relationships are across the signs! Both signs conve information in terms of the the focus of action) s(itching the operative mode bet(een them through the signs on the interface! %he right hand button denotes LL*nable +e ?uardMM as its function! %his is not an action that (ill move the user closer to achieving the goal i!e! the combination of displa ed signs does not mean (hat the user (ants to communicate to the phone! %he user is forced to loo$ else(here on the interface for a possible action that (ill move them further for(ards! %his is (here the information supplied b the signs in relation to the Navigation button fulfils the intention of =chec$ing messages! %he signs have to be organised in the correct s ntagmatic structure in order to proceed correctl ) other(ise an alternative function of the phone (ill be brought into operation!

AS/SA n: 4;) p! 7D

UserAction: Use the Navigation button to scroll do(n the choices one at a time!

SystemState: %he indicator arro( moves do(n to the ne&t heading on the list and the number at the top of the screen changes from 4 to 2! =8enu is displa ed instead of =*nable!

Figure 4./ The "enu screen stage %

%he ne&t screen -,igure @!D0 sho(s ho( the meanings of the t(o buttons interrelate (ith one another! %he function LL8ove Do(nMM is directl related to the change in state of the right hand button LLdispla Choneboo$ 8enuMM! Again this is not the meaning the user is loo$ing for! %he concurrent s ntagm created bet(een the signs on the screen and t(o button signs generates the meaning LL8ove Do(nMM) (hich in turn moves focus to the do(n arro( at the bottom of the navigation button in order to get to =8essages!

UserAction: Use the Navigation button to move do(n to =8essages! System State: %he indicator moves do(n to point at =8essages! %he number sho(s three!

Figure 4.. The "enu screen stage 3

1aving no( reached the =8essages option -,igure @!60 the right hand button becomes the focus of activit again! %he indicator arro( is pointing at the choice the user (ants to ma$e! Also) it is apparent that if the Navigation button is pressed again the user (ill go too far do(n the list! B using *cos formula at this level it can be seen ho( the various functional meanings of the signs are created as the user moves through the interaction! 8oreover) it is possible to see ho( meaning is made through understanding the relationship of the signs that are constructed (ithin the interface! In other (ords) it is the users s ntagmatic structuring of the signs in the interface that is a central part of interaction!

!.1.1 Comparisons Across hree Phones Loo$ing at "ust one sign it is possible to see ho( *cos model can act as a map of its various functions across the different conte&ts and circumstances (ithin the interactive possibilities of the phone! ,igure @!43 sho(s the map of related functions that the right hand button has in the various stages of the =Chec$ 8essages interaction! %he structure of the semantic tree becomes more comple& as various configurations of signs are revealed according to the differing Circumstances and Conte&ts that occur throughout the interaction! In all the phones e&amined) similar screen states and relationships bet(een the signs in the interface (ere encountered!

Figure 4.10 : (unctiona$ "ode$ o( "eaning (or the Panasonic GD35 Se$ect #utton

AS/SA n: 4;) p! 76

If this mapping (ere to continue over ever interaction that the button has a relationship (ith) it could be sho(n "ust ho( comple& the construction of meaning (ith regard to this sign is (ithin the device as a (hole! %his can be seen as a map of the operation of the button from (hich a concept is derived about its operation! In general terms) (hile there are a number of different denoted meanings related to this button a close loo$ at them reveals that a lot of them have a similar meaning! 9hat is learnt about the button is that it operates across the signs in the interface as a =Select button! %he abstract concept derived from its operation is one that allo(s ou to ma$e a selection choice based on the comple& arrangement of interface signs at an moment throughout the interaction! %hereb (e have some notion related to the sign about the concept (ith (hich it operates!

Figure 4.11 : (unctiona$ "ode$ (or the Nokia 5110 Se$ect #utton

In the other phones -,igure @!440 similar patterns of meaning structures emerge through anal sing interactions (ithin their specific s stem of signs! %he middle button on the <443 and t(o buttons on the ;243 have blue lines across them! Despite the differences in the position) si.e and form of s mbol on these buttons) the behave "ust li$e the select buttons on the ?D7<!

Figure 4.1% : (unctiona$ "ode$ o( the GD35 Na2igation #utton

In the same (a other buttons across the phones behave in similar (a s! ,or e&ample) the arro( buttons mentioned earlier ,igure @!42! Although on all the phones the appear graphicall distinct the not onl have similar semantic fields) but in practice the actuall operate in ver similar (a s across the phones ,igure @!47!

Figure 4.13 : (unctiona$ 1ode$ o( the Nokia 6150 na2igation #utton

Both the ?D7< and the <443 have a button mar$ed (ith a =C but the ;243 does not! %he function of this button is not immediatel apparent but after some

investigation it becomes clear that the general function of the button is as a =Cancel button -,igure @!4@0!

Figure 4.14 The conte3tua$ (unctiona$ re$ationships o( the 89, #utton

AS/SA n: 4;) p! @3

It operates in different (a s depending on the length of time it is held do(n) but in general its function is to erase functions that have "ust been performed) returning to the main screen! %he function of this button is identical on both phones ,igure @!4<!

Figure 4.15 1etas "#o$ic aspects o( Se$ect signs

Different phones use different s mbols et (hen one is understood in one phone the principles in application are transferable across phones! 9hile not strictl across domain) this is essentiall the principle of metaphor at (or$) (here it becomes possible to substitute the signifier from one phone for the signifier on another (hile maintaining the same functional meaning -see ,igure @!4< and ,igure @!4;0!

Figure 4.16 1etas "#o$ic aspects o( na2igation signs

!.! Findings
Consideration of the first part of this stud reveals ho( some signifiers carr a certain amount of cultural baggage (ith them) (hich offer a host of potential meanings in the form of a semantic field! ?iven a certain conte&t and/or circumstance onl certain meanings become appropriate! %his is ho( coding and decoding (or$s! *ssentiall *cos theories) the revised +, model in particular) offer an opportunit to articulate notions of decoding in relation to denotations and connotations (hich are dependent on the conte&t and circumstance in (hich the are encountered! Indeed) as the second part of this stud sho(s) ne( signs often have no cultural frame of reference) the are under>coded) and (or$ commences to establish (hat the mean in relation to the signs around them! %he second part of the stud concentrates on e&ploring the possible meanings of a sign in relation to the changing conte&ts and circumstances that occur during interacting (ith the phone! At each stage of interaction the relationships bet(een signs and the resultant meanings are laid bare! %his section in particular highlights ho( the concepts of s ntagmatic structuring are articulated during an interaction! *ach screen in itself is a grouping of concurrent signs) (hich are based on the internal paradigmatic structuring of functionalit (ithin the phone! %he concurrent arrangements of signs in screens allude to interactive possibilities) (hile the resultant changes in those signs become manifest through the se'uential nature of interaction! 8anipulating the phones controls to produce the correct se'uential s ntagmatic relationships out of the concurrent arrangement of signs on each screen establishes an end goal! 9hile this is onl a ver short tas$) this stud sho(s ho( comple& these relationships are) as (ell as ho( time consuming it might be to anal se the (hole interface! 9ith this in mind section @!4!7 considers ho( the semantic fields of signs become e&tended through establishing a (or$ing concept of a sign (ithin a particular set of conte&ts and circumstances! 8oreover) relationships bet(een signifiers can be established (hen it becomes apparent that the are associated (ith the same (or$ing concept A difficult in implementing *cos formula to interactive interfaces (ould appear to be that it is far more difficult to establish the conte&t and circumstance of a sign (ithin an interface due to the hidden depths of its functionalit (ithin the

operation of the device as a (hole! Conte&t and circumstance in (ritten te&ts are constituted b the other (ords that surround the (ord or phrase being anal sed! 9ithin an interactive interface this remains true) but there ma also be man hidden operations that alter the meanings of a sign that onl become apparent over time!

AS/SA n: 4;) p! @4


*ach of the anal tic techni'ues used in this chapter are different) but each has been used in an attempt to appl some aspect of semiotic theor to the anal sis and interpretation of interface elements! 9hile each one has its o(n particular strengths and (ea$nesses) the important thing to consider is ho( some relevant aspects of semiotic theor have been identified and e&plored) in relation to the interactive s stems and ne( media! ,or e&ample) it is 'uite apparent in the product anal sis) (hich focuses on a sign t pe) that icons) indices and s mbols are evident across all three phones! %his t pe of anal sis (as able to identif subtle differences in the construction of these sign t pes) (hich (hile often loo$ing different conve ed the same sorts of meanings across all three phones! All three studies provide evidence of the concurrent arrangement of signs across the three phones! %he first anal sis identifies not onl similar s ntagmatic structures of signs across each phone) but also different signs (ith similar meanings) (hich are arranged differentl in the interface! Similarl ) the visual anal sis also uncovers these differences) along (ith the fundamental organisation of mobile phone forms) such as the hierarchical visual structure of screen above control $e s! 1o(ever) this t pe of anal sis is fairl limited in relation to uncovering aspects of meaning over and above the structure of the interface! In the third anal sis) again concurrent s ntagmatic relationships are discussed) but more importantl ) the third anal sis of interaction uncovers the (a in (hich signs are manipulated se'uentiall in order to establish meanings! Interaction across all three phones is reliant on the abilit of a user to interpret and manipulate signs in both concurrent and se'uential relationships! %hrough e&ploring semantic fields along (ith the conte&tual and circumstantial relationships that establish denotations and connotations) it becomes apparent that man signs in the phones interfaces have man potential meanings! 9hile the anal sis predominantl e&plores ho( a signs

functional meaning is established through interaction) attention is dra(n to the place of this meaning in relation to a (ider cultural conte&t! %his particular aspect of the anal sis highlights the role of the reader as an interactive interpreter of the interface! 8eanings are derived through the manipulation of signs (ithin the structure of the interface as (ell as the realm of potential meanings brought to the interaction b the user! Obviousl in interfaces) such as these) that attempt to communicate their functionalit through signs and s mbols) there is a real potential for misunderstanding! Signs do not al(a s conve a one to one relationship (ith functionalit and it is onl through learning the language of the interface signs that the full potential of the phone can be unloc$ed! Clearl ) some elements of these three studies highlight this problem and offer the possibilit to identif combinations of signs that might prove to be confusing! %hus) this e&ploration of semiotic anal sis highlights thatG It is not al(a s clear (hat signs initiall mean! 8an signs in interfaces are culturall defined and m bring confusing multiple meanings (ith them from e&ternal domains! Similarl ) some signs are =phone specific and are thus under coded) ma$ing it difficult to understand the purpose the denote! %he functionalit of the devices are e&pressed b signs that have to be learned! %he under coding of some signs in the interface means that phone interfaces have their o(n language) ma$ing it difficult for novice users to pic$ up and interpret! Signs tend to be s mbolic rather than iconic or inde&ical! Opa'ue rather than transparent! It is the se'uential process of interacting (ith the d namic interface that creates meaning) even (hen the interface signs arent full understood! It is the concurrent and se'uential structuring of elements b the user that allo(s for meaning to be produced and functions to be communicated! %he designer and user are both authors and readers of interactive te&ts!


Icons) indices and s mbols) as (ell as concurrent and se'uential s ntagms) are all evident in each of the phone studies presented here! ,or e&ample) (e see the abstraction of batter life and signal strength into inde&ical signs that communicate the phones state! 8oreover) in ne(er versions of these phones) concepts such as address boo$s and tools for personalising our phone are abstracted and represented b icons such as a picture of an address boo$ or scre(drivers and spanners respectivel ! %herefore) semiotic theor does indeed connect the technologies of ne( media to the criti'ue of older media! In doing so) this semiotic approach provides useful terminolog that helps to articulate the problems conve ing meaning! It is (ithout doubt ver useful to thin$ of ne( interactive media from this perspective!

AS/SA n: 4;) p! @2

,urthermore) ne( phones continue the progression of convergent technolog ) (here sign s stems from different domains of functionalit are integrated into one device! ,or e&ample man ne( phones contain signs s stems that re'uire the understanding of cameras) media pla ers and Internet connection) as (ell as the no( familiar te&ting and calling options! %his convergence of functionalit leads to more complicated interfaces and thus more need to develop understandable and meaningful signs to represent functionalit ! A semiotics of interactive media must address this problem b establishing (hat $ind of signs it hopes to identif and ho( people ma$e sense of interacting (ith them! Older semiotic theor ) in the most part) is aimed at anal sing =static te&ts! I!e! te&ts constructed b an author (here the constituent parts have been organi.ed into a structure that does not change over time! As seen in the studies presented here) in interactive media) this is not the case! Although designers are still responsible for organising the structure of soft(are applications) the ver nature of (hat ma$es them interactive i!e! the introduction of the agenc of the user) ma$es semiotic anal sis (ith traditional static methods problematic! %he possibilit for personalisation and self> organisation) as controlled b the user in some applications) confounds the traditional static semiotic approach even further! A semiotics of ne( media has to be able to cope (ith d namic te&ts that alter over time as users interact (ith interfaces and content! %he three semiotic anal ses presented here do not individuall provide e&amples of relevant semiotic theor that is useful for anal sing interactive s stems! None of them are singularl effective in describing the phones! 1o(ever) each of the anal sis provides evidence for at least one aspect of relevant semiotic theor (ith significant crossover bet(een them all! 9hen the are considered together) the possibilit of an integrated semiotic theor from older media in relation to anal sing convergent media artefacts becomes an attractive proposition! Using semiotic anal sis) in this (a ) highlights that elements of older semiotic approaches) (hile useful) are not enough in themselves! In order to understand interactions (ith ne( media) the relevant aspects of semiotic theor must be combined together) in such a (a ) as to produce a semiotics of ne( media that is capable of articulating its specific characteristics! %his is particularl relevant for interactive s stems outside the (or$ domain) (here there is a great deal more speculation surrounding interpretation and meaning) e!g! interactive %E) games) interactive art(or$!


Andersen) C! B! -46630 A Theory of Computer Semiotics ) Cambridge) Cambridge Universit Cress! Andersen) Ceter BQgh) -23340 9hat Semiotics can and cannot do for 1CI) Knowledge-Based Systems) Eolume 4@) Issue D) 4 December 2334) pp! @46> @2@! Barthes) J! -46AA0 Image, usic, Te!t) ,ontana Cress) London! Ben on) D! -466@0 A ,unctional 8odel of Interacting S stemsG A semiotic Approach! In K! 1! Connell / *! A! *dmonds -*ds!0) CSC" and Artificial Intelligence) LondonG Springer Eerlag) pp! 43<>42<! Ben on) D! -23340 %he ne( 1CIP Navigation of information space) KnowledgeBased Systems) Eolume 4@) Issue D) 4 December 2334) pp! @2<>@73! Bolter) K! D! and ?rusin) J! -46660 Remediation# $nderstanding %ew 8I% Cress) Cambridge) 8assachusetts! edia) %he

B rne) 8!D! -23370 8ental 8odels) in %he 1uman>Computer Interaction 1andboo$G *volving %echnologies and *merging Applications) K!A! Kac$o and A Sears -*ds0) La(rence *arlbaum Associates) Ne( Kerse ) pp! 6A>44A de 1aan) ?! -466@0 An *%A?>based Approach to the Design of User Interfaces! InG %auber) 8!K!) %raunmReller) J! and +aplan) S! -eds!0 Croceedings Interdisciplinar approaches to S stem Anal sis and Design! SchSrding) Austria) 2@>2; 8a ) 466@! de 1aan) ?! -466A0 1o( to Coo$ *%A? and Jelated DishesG Uses of a Notational Language for User>Interface Design! Cognitive S stems @) 7>@) 7<7>7A6! de 1aan) ?! -23330 &TA', a (ormal odel of Competence Knowledge for $ser Interface )esign! Doctoral thesis! Department of 8athematics and Computer Science) ,ree Universit Amsterdam) October 2333! de Sou.a) C! S!) Barbosa) S! D! K! and Crates) J! O!) -23340 A semiotic engineering approach to user interface design) Knowledge-Based Systems) Eolume 4@) Issue D) 4 December 2334) Cages @;4>@;! *co) U -23330 +ant and the Clat pusG *ssa s on Language and Cognition) Eintage) London!

AS/SA n: 4;) p! @7

*co) U! -46A;0 A Theory of Semiotics) Indiana Universit Cress) Indiana *co) U! -46D;0 ,unction and SignG Semiotics of Architecture! In 8! ?ottdiener / A! Lagopoulos -*ds!0) The City and the Sign) Columbia Universit Cress) Ne( Tor$! 1allida ) 8! A! +! -46AD0 *anguage as social semiotic, The social interpretation of language and meaning) *d(ard Arnold) London!

1"elmslev) L! -46;40 +rolegomena to a Theory of *anguage! 8adisonG Universit of 9isconcin Cress! +arat) K! -23370 8ental 8odels) in %he 1uman>Computer Interaction 1andboo$G *volving %echnologies and *merging Applications) K!A! Kac$o and A Sears -*ds0) La(rence *arlbaum Associates) Ne( Kerse ) pp! 44<2>44;@! +ieras) D! -23370 8ental 8odels) in %he 1uman>Computer Interaction 1andboo$G *volving %echnologies and *merging Applications) K!A! Kac$o and A Sears -*ds0) La(rence *arlbaum Associates) Ne( Kerse ) pp! 4476>44<4! +ita"ima) 8! and Colson) C! ?! -46620 U A +rocess odel of )isplay-Based ,uman Computer Interaction-U C1I 4662 Jesearch S mposium! +ita"ima) 8! and Colson) C! ?! -466<0! 8easuring the ?ulf of *valuation in Displa > Based 1CI! A position paper presented at the (or$shop on Cognitive Architecture and 1CI in C1IV6< +ress) ?! and van Leeu(en) %! -466;0 Reading Images .The grammar of visual design/) Joutledge) London! La$off) ?! and Kohnson) 8! -46D30 Cress) Chicago! etaphors "e *ive By) Universit of Chicago

La$off) ?! and Kohnson) 8! -46660 +hilosophy of the (lesh) Basic Boo$s) Ne( Tor$G 8cCullogh) 8! -466;0 A0stracting Craft# the practiced digital hand) %he 8I% Cress) Cambridge 8assachusetts! Nadin) 8! -23340 One cannot not interact) Knowledge-Based Systems) Eolume 4@) Issue D) 4 December 2334) Cages @7A>@@3! Norman) D! -46DD0! The )esign of &veryday Things! %he 8I% Cress) London) *ngland! ONeill) S! K! -233<0 &!ploring a Semiotics of %ew Universit ) *dinburgh! edia) Doctoral %hesis) Napier

OVNeill) S! K! and Ben on) D! J! -2337a0! An &!ploration of a Semiotic odel of Interaction Through Interactive edia- Caper presented at the the International 1CI) Arts and 1umanities (or$shop) 24st Kul 2337) Universit of Tor$) Tor$) U+! OVNeill) S! and Ben on) D! -2337b0! A Semiotic Approach to Investingating +resenceCaper presented at the COSI?N>2337) 8iddlesborough! OVNeill) S!) Ben on) D! J!) and %urner) S! J! -23320! Semiotics and Interaction Analysis- Caper presented at the *CC* 44) Catania) Sicil ! OVNeill) S!) Ben on) D! J!) and %urner) S! J! -forthcoming0! %he Semiotics of Interactive S stems! To appear in Cognition, Technology and "or1! Ceirce) C! S! -4674>46<D0 Collected +apers of Charles Sanders +eirce -Eol! 40! 1arvard Universit Cress) Cambridge! Crates J) de Sou.a C! S! and Barbosa S! D! K! -23330 A method for evaluating the communicabilit of User Interfaces) Interactions 2an3(e0) pp! 74>7D! Saussure) ,! -46;;0 Course in 'eneral *inguistics) 8c?ra( 1ill) Ne( Tor$!

van der Eeer) ?! and del Carmen Cuerta 8! -23370 8ental 8odels) in %he 1uman>Computer Interaction 1andboo$G *volving %echnologies and *merging Applications) K!A! Kac$o and A Sears -*ds0) La(rence *arlbaum Associates) Ne( Kerse ) pp! <2>D3! Eihma) S! -466<0 +roducts as Representations - a semiotic and aesthetic study of design products! 1elsin$iG Universit of Art and Design 1elsin$i! 9esterlund) B! -23320 (orm is (unction- Caper presented at the DIS2332) London!

E-mail the editors Pour crire la rdaction

2005, Applied Semiotics / Smiotique applique